2005-01-07 02:29:27 +00:00
|
|
|
/*-
|
2017-11-20 19:43:44 +00:00
|
|
|
* SPDX-License-Identifier: BSD-3-Clause
|
|
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Copyright (c) 1990 University of Utah.
|
|
|
|
|
* Copyright (c) 1991, 1993
|
|
|
|
|
* The Regents of the University of California. All rights reserved.
|
|
|
|
|
*
|
|
|
|
|
* This code is derived from software contributed to Berkeley by
|
|
|
|
|
* the Systems Programming Group of the University of Utah Computer
|
|
|
|
|
* Science Department.
|
|
|
|
|
*
|
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
2017-02-28 23:42:47 +00:00
|
|
|
* 3. Neither the name of the University nor the names of its contributors
|
1994-05-24 10:09:53 +00:00
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
|
* without specific prior written permission.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
|
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
|
*
|
|
|
|
|
* @(#)vm_pager.h 8.4 (Berkeley) 1/12/94
|
1999-08-28 01:08:13 +00:00
|
|
|
* $FreeBSD$
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Pager routine interface definition.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef _VM_PAGER_
|
|
|
|
|
#define _VM_PAGER_
|
|
|
|
|
|
1998-02-03 22:19:35 +00:00
|
|
|
#include <sys/queue.h>
|
|
|
|
|
|
2000-05-26 02:09:24 +00:00
|
|
|
TAILQ_HEAD(pagerlst, vm_object);
|
2021-05-01 01:04:04 +00:00
|
|
|
struct vnode;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2004-11-09 13:43:20 +00:00
|
|
|
typedef void pgo_init_t(void);
|
Implement global and per-uid accounting of the anonymous memory. Add
rlimit RLIMIT_SWAP that limits the amount of swap that may be reserved
for the uid.
The accounting information (charge) is associated with either map entry,
or vm object backing the entry, assuming the object is the first one
in the shadow chain and entry does not require COW. Charge is moved
from entry to object on allocation of the object, e.g. during the mmap,
assuming the object is allocated, or on the first page fault on the
entry. It moves back to the entry on forks due to COW setup.
The per-entry granularity of accounting makes the charge process fair
for processes that change uid during lifetime, and decrements charge
for proper uid when region is unmapped.
The interface of vm_pager_allocate(9) is extended by adding struct ucred *,
that is used to charge appropriate uid when allocation if performed by
kernel, e.g. md(4).
Several syscalls, among them is fork(2), may now return ENOMEM when
global or per-uid limits are enforced.
In collaboration with: pho
Reviewed by: alc
Approved by: re (kensmith)
2009-06-23 20:45:22 +00:00
|
|
|
typedef vm_object_t pgo_alloc_t(void *, vm_ooffset_t, vm_prot_t, vm_ooffset_t,
|
|
|
|
|
struct ucred *);
|
2004-11-09 13:43:20 +00:00
|
|
|
typedef void pgo_dealloc_t(vm_object_t);
|
A change to KPI of vm_pager_get_pages() and underlying VOP_GETPAGES().
o With new KPI consumers can request contiguous ranges of pages, and
unlike before, all pages will be kept busied on return, like it was
done before with the 'reqpage' only. Now the reqpage goes away. With
new interface it is easier to implement code protected from race
conditions.
Such arrayed requests for now should be preceeded by a call to
vm_pager_haspage() to make sure that request is possible. This
could be improved later, making vm_pager_haspage() obsolete.
Strenghtening the promises on the business of the array of pages
allows us to remove such hacks as swp_pager_free_nrpage() and
vm_pager_free_nonreq().
o New KPI accepts two integer pointers that may optionally point at
values for read ahead and read behind, that a pager may do, if it
can. These pages are completely owned by pager, and not controlled
by the caller.
This shifts the UFS-specific readahead logic from vm_fault.c, which
should be file system agnostic, into vnode_pager.c. It also removes
one VOP_BMAP() request per hard fault.
Discussed with: kib, alc, jeff, scottl
Sponsored by: Nginx, Inc.
Sponsored by: Netflix
2015-12-16 21:30:45 +00:00
|
|
|
typedef int pgo_getpages_t(vm_object_t, vm_page_t *, int, int *, int *);
|
2014-11-23 12:01:52 +00:00
|
|
|
typedef void pgo_getpages_iodone_t(void *, vm_page_t *, int, int);
|
A change to KPI of vm_pager_get_pages() and underlying VOP_GETPAGES().
o With new KPI consumers can request contiguous ranges of pages, and
unlike before, all pages will be kept busied on return, like it was
done before with the 'reqpage' only. Now the reqpage goes away. With
new interface it is easier to implement code protected from race
conditions.
Such arrayed requests for now should be preceeded by a call to
vm_pager_haspage() to make sure that request is possible. This
could be improved later, making vm_pager_haspage() obsolete.
Strenghtening the promises on the business of the array of pages
allows us to remove such hacks as swp_pager_free_nrpage() and
vm_pager_free_nonreq().
o New KPI accepts two integer pointers that may optionally point at
values for read ahead and read behind, that a pager may do, if it
can. These pages are completely owned by pager, and not controlled
by the caller.
This shifts the UFS-specific readahead logic from vm_fault.c, which
should be file system agnostic, into vnode_pager.c. It also removes
one VOP_BMAP() request per hard fault.
Discussed with: kib, alc, jeff, scottl
Sponsored by: Nginx, Inc.
Sponsored by: Netflix
2015-12-16 21:30:45 +00:00
|
|
|
typedef int pgo_getpages_async_t(vm_object_t, vm_page_t *, int, int *, int *,
|
2014-11-23 12:01:52 +00:00
|
|
|
pgo_getpages_iodone_t, void *);
|
2004-11-09 13:43:20 +00:00
|
|
|
typedef void pgo_putpages_t(vm_object_t, vm_page_t *, int, int, int *);
|
|
|
|
|
typedef boolean_t pgo_haspage_t(vm_object_t, vm_pindex_t, int *, int *);
|
Add a new populate() pager method and extend device pager ops vector
with cdev_pg_populate() to provide device drivers access to it. It
gives drivers fine control of the pages ownership and allows drivers
to implement arbitrary prefault policies.
The populate method is called on a page fault and is supposed to
populate the vm object with the page at the fault location and some
amount of pages around it, at pager's discretion. VM provides the
pager with the hints about current range of the object mapping, to
avoid instantiation of immediately unused pages, if pager decides so.
Also, VM passes the fault type and map entry protection to the pager,
allowing it to force the optimal required ownership of the mapped
pages.
Installed pages must contiguously fill the returned region, be fully
valid and exclusively busied. Of course, the pages must be compatible
with the object' type.
After populate() successfully returned, VM fault handler installs as
many instantiated pages into the process page tables as it sees
reasonable, while still obeying the correct semantic for COW and vm
map locking.
The method is opt-in, pager sets OBJ_POPULATE flag to indicate that
the method can be called. If pager' vm objects can be shadowed, pager
must implement the traditional getpages() method in addition to the
populate(). Populate() might fall back to the getpages() on per-call
basis as well, by returning VM_PAGER_BAD error code.
For now for device pagers, the populate() method is only allowed to be
used by the managed device pagers, but the limitation is only made
because there is no unmanaged fault handlers which could use it right
now.
KPI designed together with, and reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
2016-12-08 11:26:11 +00:00
|
|
|
typedef int pgo_populate_t(vm_object_t, vm_pindex_t, int, vm_prot_t,
|
|
|
|
|
vm_pindex_t *, vm_pindex_t *);
|
2004-11-09 13:43:20 +00:00
|
|
|
typedef void pgo_pageunswapped_t(vm_page_t);
|
2019-09-03 20:31:48 +00:00
|
|
|
typedef void pgo_writecount_t(vm_object_t, vm_offset_t, vm_offset_t);
|
2021-05-01 00:14:48 +00:00
|
|
|
typedef void pgo_set_writeable_dirty_t(vm_object_t);
|
2021-05-01 00:43:08 +00:00
|
|
|
typedef bool pgo_mightbedirty_t(vm_object_t);
|
2021-05-01 01:04:04 +00:00
|
|
|
typedef void pgo_getvp_t(vm_object_t object, struct vnode **vpp,
|
|
|
|
|
bool *vp_heldp);
|
2021-05-01 17:12:32 +00:00
|
|
|
typedef void pgo_freespace_t(vm_object_t object, vm_pindex_t start,
|
|
|
|
|
vm_size_t size);
|
2004-11-09 13:43:20 +00:00
|
|
|
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
struct pagerops {
|
2021-05-07 17:46:51 +00:00
|
|
|
int pgo_kvme_type;
|
2014-11-14 18:15:35 +00:00
|
|
|
pgo_init_t *pgo_init; /* Initialize pager. */
|
|
|
|
|
pgo_alloc_t *pgo_alloc; /* Allocate pager. */
|
|
|
|
|
pgo_dealloc_t *pgo_dealloc; /* Disassociate. */
|
|
|
|
|
pgo_getpages_t *pgo_getpages; /* Get (read) page. */
|
2014-11-23 12:01:52 +00:00
|
|
|
pgo_getpages_async_t *pgo_getpages_async; /* Get page asyncly. */
|
2014-11-14 18:15:35 +00:00
|
|
|
pgo_putpages_t *pgo_putpages; /* Put (write) page. */
|
|
|
|
|
pgo_haspage_t *pgo_haspage; /* Query page. */
|
Add a new populate() pager method and extend device pager ops vector
with cdev_pg_populate() to provide device drivers access to it. It
gives drivers fine control of the pages ownership and allows drivers
to implement arbitrary prefault policies.
The populate method is called on a page fault and is supposed to
populate the vm object with the page at the fault location and some
amount of pages around it, at pager's discretion. VM provides the
pager with the hints about current range of the object mapping, to
avoid instantiation of immediately unused pages, if pager decides so.
Also, VM passes the fault type and map entry protection to the pager,
allowing it to force the optimal required ownership of the mapped
pages.
Installed pages must contiguously fill the returned region, be fully
valid and exclusively busied. Of course, the pages must be compatible
with the object' type.
After populate() successfully returned, VM fault handler installs as
many instantiated pages into the process page tables as it sees
reasonable, while still obeying the correct semantic for COW and vm
map locking.
The method is opt-in, pager sets OBJ_POPULATE flag to indicate that
the method can be called. If pager' vm objects can be shadowed, pager
must implement the traditional getpages() method in addition to the
populate(). Populate() might fall back to the getpages() on per-call
basis as well, by returning VM_PAGER_BAD error code.
For now for device pagers, the populate() method is only allowed to be
used by the managed device pagers, but the limitation is only made
because there is no unmanaged fault handlers which could use it right
now.
KPI designed together with, and reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
2016-12-08 11:26:11 +00:00
|
|
|
pgo_populate_t *pgo_populate; /* Bulk spec pagein. */
|
2014-11-14 18:00:00 +00:00
|
|
|
pgo_pageunswapped_t *pgo_pageunswapped;
|
2019-09-03 20:31:48 +00:00
|
|
|
pgo_writecount_t *pgo_update_writecount;
|
|
|
|
|
pgo_writecount_t *pgo_release_writecount;
|
2021-05-01 00:14:48 +00:00
|
|
|
pgo_set_writeable_dirty_t *pgo_set_writeable_dirty;
|
2021-05-01 00:43:08 +00:00
|
|
|
pgo_mightbedirty_t *pgo_mightbedirty;
|
2021-05-01 01:04:04 +00:00
|
|
|
pgo_getvp_t *pgo_getvp;
|
2021-05-01 17:12:32 +00:00
|
|
|
pgo_freespace_t *pgo_freespace;
|
1994-05-24 10:09:53 +00:00
|
|
|
};
|
|
|
|
|
|
2021-05-03 21:58:07 +00:00
|
|
|
extern const struct pagerops defaultpagerops;
|
|
|
|
|
extern const struct pagerops swappagerops;
|
|
|
|
|
extern const struct pagerops vnodepagerops;
|
|
|
|
|
extern const struct pagerops devicepagerops;
|
|
|
|
|
extern const struct pagerops physpagerops;
|
|
|
|
|
extern const struct pagerops sgpagerops;
|
|
|
|
|
extern const struct pagerops mgtdevicepagerops;
|
|
|
|
|
extern const struct pagerops swaptmpfspagerops;
|
2003-08-03 09:27:39 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
|
* get/put return values
|
|
|
|
|
* OK operation was successful
|
|
|
|
|
* BAD specified data was out of the accepted range
|
|
|
|
|
* FAIL specified data was in range, but doesn't exist
|
|
|
|
|
* PEND operations was initiated but not completed
|
|
|
|
|
* ERROR error while accessing data that is in range and exists
|
|
|
|
|
* AGAIN temporary resource shortage prevented operation from happening
|
|
|
|
|
*/
|
|
|
|
|
#define VM_PAGER_OK 0
|
|
|
|
|
#define VM_PAGER_BAD 1
|
|
|
|
|
#define VM_PAGER_FAIL 2
|
|
|
|
|
#define VM_PAGER_PEND 3
|
|
|
|
|
#define VM_PAGER_ERROR 4
|
|
|
|
|
#define VM_PAGER_AGAIN 5
|
|
|
|
|
|
2002-12-28 21:03:42 +00:00
|
|
|
#define VM_PAGER_PUT_SYNC 0x0001
|
|
|
|
|
#define VM_PAGER_PUT_INVAL 0x0002
|
2016-11-23 17:53:07 +00:00
|
|
|
#define VM_PAGER_PUT_NOREUSE 0x0004
|
2002-12-28 21:03:42 +00:00
|
|
|
#define VM_PAGER_CLUSTER_OK 0x0008
|
This mega-commit is meant to fix numerous interrelated problems. There
has been some bitrot and incorrect assumptions in the vfs_bio code. These
problems have manifest themselves worse on NFS type filesystems, but can
still affect local filesystems under certain circumstances. Most of
the problems have involved mmap consistancy, and as a side-effect broke
the vfs.ioopt code. This code might have been committed seperately, but
almost everything is interrelated.
1) Allow (pmap_object_init_pt) prefaulting of buffer-busy pages that
are fully valid.
2) Rather than deactivating erroneously read initial (header) pages in
kern_exec, we now free them.
3) Fix the rundown of non-VMIO buffers that are in an inconsistent
(missing vp) state.
4) Fix the disassociation of pages from buffers in brelse. The previous
code had rotted and was faulty in a couple of important circumstances.
5) Remove a gratuitious buffer wakeup in vfs_vmio_release.
6) Remove a crufty and currently unused cluster mechanism for VBLK
files in vfs_bio_awrite. When the code is functional, I'll add back
a cleaner version.
7) The page busy count wakeups assocated with the buffer cache usage were
incorrectly cleaned up in a previous commit by me. Revert to the
original, correct version, but with a cleaner implementation.
8) The cluster read code now tries to keep data associated with buffers
more aggressively (without breaking the heuristics) when it is presumed
that the read data (buffers) will be soon needed.
9) Change to filesystem lockmgr locks so that they use LK_NOPAUSE. The
delay loop waiting is not useful for filesystem locks, due to the
length of the time intervals.
10) Correct and clean-up spec_getpages.
11) Implement a fully functional nfs_getpages, nfs_putpages.
12) Fix nfs_write so that modifications are coherent with the NFS data on
the server disk (at least as well as NFS seems to allow.)
13) Properly support MS_INVALIDATE on NFS.
14) Properly pass down MS_INVALIDATE to lower levels of the VM code from
vm_map_clean.
15) Better support the notion of pages being busy but valid, so that
fewer in-transit waits occur. (use p->busy more for pageouts instead
of PG_BUSY.) Since the page is fully valid, it is still usable for
reads.
16) It is possible (in error) for cached pages to be busy. Make the
page allocation code handle that case correctly. (It should probably
be a printf or panic, but I want the system to handle coding errors
robustly. I'll probably add a printf.)
17) Correct the design and usage of vm_page_sleep. It didn't handle
consistancy problems very well, so make the design a little less
lofty. After vm_page_sleep, if it ever blocked, it is still important
to relookup the page (if the object generation count changed), and
verify it's status (always.)
18) In vm_pageout.c, vm_pageout_clean had rotted, so clean that up.
19) Push the page busy for writes and VM_PROT_READ into vm_pageout_flush.
20) Fix vm_pager_put_pages and it's descendents to support an int flag
instead of a boolean, so that we can pass down the invalidate bit.
1998-03-07 21:37:31 +00:00
|
|
|
|
1999-12-29 05:07:58 +00:00
|
|
|
#ifdef _KERNEL
|
1997-10-12 20:26:33 +00:00
|
|
|
|
2021-05-03 21:58:07 +00:00
|
|
|
extern const struct pagerops *pagertab[] __read_mostly;
|
2013-06-28 03:51:20 +00:00
|
|
|
extern struct mtx_padalign pbuf_mtx;
|
1995-07-29 11:44:31 +00:00
|
|
|
|
Make MAXPHYS tunable. Bump MAXPHYS to 1M.
Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.
Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.
Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.
Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.
Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225
2020-11-28 12:12:51 +00:00
|
|
|
/*
|
|
|
|
|
* Number of pages that pbuf buffer can store in b_pages.
|
|
|
|
|
* It is +1 to allow for unaligned data buffer of maxphys size.
|
|
|
|
|
*/
|
|
|
|
|
#define PBUF_PAGES (atop(maxphys) + 1)
|
|
|
|
|
|
Implement global and per-uid accounting of the anonymous memory. Add
rlimit RLIMIT_SWAP that limits the amount of swap that may be reserved
for the uid.
The accounting information (charge) is associated with either map entry,
or vm object backing the entry, assuming the object is the first one
in the shadow chain and entry does not require COW. Charge is moved
from entry to object on allocation of the object, e.g. during the mmap,
assuming the object is allocated, or on the first page fault on the
entry. It moves back to the entry on forks due to COW setup.
The per-entry granularity of accounting makes the charge process fair
for processes that change uid during lifetime, and decrements charge
for proper uid when region is unmapped.
The interface of vm_pager_allocate(9) is extended by adding struct ucred *,
that is used to charge appropriate uid when allocation if performed by
kernel, e.g. md(4).
Several syscalls, among them is fork(2), may now return ENOMEM when
global or per-uid limits are enforced.
In collaboration with: pho
Reviewed by: alc
Approved by: re (kensmith)
2009-06-23 20:45:22 +00:00
|
|
|
vm_object_t vm_pager_allocate(objtype_t, void *, vm_ooffset_t, vm_prot_t,
|
|
|
|
|
vm_ooffset_t, struct ucred *);
|
2002-03-19 22:20:14 +00:00
|
|
|
void vm_pager_bufferinit(void);
|
|
|
|
|
void vm_pager_deallocate(vm_object_t);
|
A change to KPI of vm_pager_get_pages() and underlying VOP_GETPAGES().
o With new KPI consumers can request contiguous ranges of pages, and
unlike before, all pages will be kept busied on return, like it was
done before with the 'reqpage' only. Now the reqpage goes away. With
new interface it is easier to implement code protected from race
conditions.
Such arrayed requests for now should be preceeded by a call to
vm_pager_haspage() to make sure that request is possible. This
could be improved later, making vm_pager_haspage() obsolete.
Strenghtening the promises on the business of the array of pages
allows us to remove such hacks as swp_pager_free_nrpage() and
vm_pager_free_nonreq().
o New KPI accepts two integer pointers that may optionally point at
values for read ahead and read behind, that a pager may do, if it
can. These pages are completely owned by pager, and not controlled
by the caller.
This shifts the UFS-specific readahead logic from vm_fault.c, which
should be file system agnostic, into vnode_pager.c. It also removes
one VOP_BMAP() request per hard fault.
Discussed with: kib, alc, jeff, scottl
Sponsored by: Nginx, Inc.
Sponsored by: Netflix
2015-12-16 21:30:45 +00:00
|
|
|
int vm_pager_get_pages(vm_object_t, vm_page_t *, int, int *, int *);
|
|
|
|
|
int vm_pager_get_pages_async(vm_object_t, vm_page_t *, int, int *, int *,
|
2015-06-17 22:44:27 +00:00
|
|
|
pgo_getpages_iodone_t, void *);
|
2002-03-19 22:20:14 +00:00
|
|
|
void vm_pager_init(void);
|
|
|
|
|
vm_object_t vm_pager_object_lookup(struct pagerlst *, void *);
|
1999-01-21 08:29:12 +00:00
|
|
|
|
1999-01-24 02:32:15 +00:00
|
|
|
static __inline void
|
2021-05-01 00:28:39 +00:00
|
|
|
vm_pager_put_pages(vm_object_t object, vm_page_t *m, int count, int flags,
|
|
|
|
|
int *rtvals)
|
|
|
|
|
{
|
2013-03-09 02:32:23 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
1999-01-24 02:32:15 +00:00
|
|
|
(*pagertab[object->type]->pgo_putpages)
|
|
|
|
|
(object, m, count, flags, rtvals);
|
1999-01-21 08:29:12 +00:00
|
|
|
}
|
|
|
|
|
|
2000-03-27 21:33:32 +00:00
|
|
|
/*
|
|
|
|
|
* vm_pager_haspage
|
|
|
|
|
*
|
|
|
|
|
* Check to see if an object's pager has the requested page. The
|
|
|
|
|
* object's pager will also set before and after to give the caller
|
|
|
|
|
* some idea of the number of pages before and after the requested
|
|
|
|
|
* page can be I/O'd efficiently.
|
|
|
|
|
*
|
2005-05-18 22:08:52 +00:00
|
|
|
* The object must be locked.
|
2000-03-27 21:33:32 +00:00
|
|
|
*/
|
1999-01-21 08:29:12 +00:00
|
|
|
static __inline boolean_t
|
2021-05-01 00:28:39 +00:00
|
|
|
vm_pager_has_page(vm_object_t object, vm_pindex_t offset, int *before,
|
|
|
|
|
int *after)
|
|
|
|
|
{
|
2001-05-19 01:28:09 +00:00
|
|
|
boolean_t ret;
|
|
|
|
|
|
2019-08-19 22:25:28 +00:00
|
|
|
VM_OBJECT_ASSERT_LOCKED(object);
|
2001-05-19 01:28:09 +00:00
|
|
|
ret = (*pagertab[object->type]->pgo_haspage)
|
|
|
|
|
(object, offset, before, after);
|
|
|
|
|
return (ret);
|
1999-01-21 08:29:12 +00:00
|
|
|
}
|
|
|
|
|
|
Add a new populate() pager method and extend device pager ops vector
with cdev_pg_populate() to provide device drivers access to it. It
gives drivers fine control of the pages ownership and allows drivers
to implement arbitrary prefault policies.
The populate method is called on a page fault and is supposed to
populate the vm object with the page at the fault location and some
amount of pages around it, at pager's discretion. VM provides the
pager with the hints about current range of the object mapping, to
avoid instantiation of immediately unused pages, if pager decides so.
Also, VM passes the fault type and map entry protection to the pager,
allowing it to force the optimal required ownership of the mapped
pages.
Installed pages must contiguously fill the returned region, be fully
valid and exclusively busied. Of course, the pages must be compatible
with the object' type.
After populate() successfully returned, VM fault handler installs as
many instantiated pages into the process page tables as it sees
reasonable, while still obeying the correct semantic for COW and vm
map locking.
The method is opt-in, pager sets OBJ_POPULATE flag to indicate that
the method can be called. If pager' vm objects can be shadowed, pager
must implement the traditional getpages() method in addition to the
populate(). Populate() might fall back to the getpages() on per-call
basis as well, by returning VM_PAGER_BAD error code.
For now for device pagers, the populate() method is only allowed to be
used by the managed device pagers, but the limitation is only made
because there is no unmanaged fault handlers which could use it right
now.
KPI designed together with, and reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
2016-12-08 11:26:11 +00:00
|
|
|
static __inline int
|
|
|
|
|
vm_pager_populate(vm_object_t object, vm_pindex_t pidx, int fault_type,
|
|
|
|
|
vm_prot_t max_prot, vm_pindex_t *first, vm_pindex_t *last)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
MPASS((object->flags & OBJ_POPULATE) != 0);
|
|
|
|
|
MPASS(pidx < object->size);
|
2020-02-28 16:05:18 +00:00
|
|
|
MPASS(blockcount_read(&object->paging_in_progress) > 0);
|
Add a new populate() pager method and extend device pager ops vector
with cdev_pg_populate() to provide device drivers access to it. It
gives drivers fine control of the pages ownership and allows drivers
to implement arbitrary prefault policies.
The populate method is called on a page fault and is supposed to
populate the vm object with the page at the fault location and some
amount of pages around it, at pager's discretion. VM provides the
pager with the hints about current range of the object mapping, to
avoid instantiation of immediately unused pages, if pager decides so.
Also, VM passes the fault type and map entry protection to the pager,
allowing it to force the optimal required ownership of the mapped
pages.
Installed pages must contiguously fill the returned region, be fully
valid and exclusively busied. Of course, the pages must be compatible
with the object' type.
After populate() successfully returned, VM fault handler installs as
many instantiated pages into the process page tables as it sees
reasonable, while still obeying the correct semantic for COW and vm
map locking.
The method is opt-in, pager sets OBJ_POPULATE flag to indicate that
the method can be called. If pager' vm objects can be shadowed, pager
must implement the traditional getpages() method in addition to the
populate(). Populate() might fall back to the getpages() on per-call
basis as well, by returning VM_PAGER_BAD error code.
For now for device pagers, the populate() method is only allowed to be
used by the managed device pagers, but the limitation is only made
because there is no unmanaged fault handlers which could use it right
now.
KPI designed together with, and reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
2016-12-08 11:26:11 +00:00
|
|
|
return ((*pagertab[object->type]->pgo_populate)(object, pidx,
|
|
|
|
|
fault_type, max_prot, first, last));
|
|
|
|
|
}
|
|
|
|
|
|
1999-01-21 08:29:12 +00:00
|
|
|
/*
|
|
|
|
|
* vm_pager_page_unswapped
|
|
|
|
|
*
|
2005-05-18 22:08:52 +00:00
|
|
|
* Destroy swap associated with the page.
|
1999-01-21 08:29:12 +00:00
|
|
|
*
|
2003-08-06 10:51:40 +00:00
|
|
|
* XXX: A much better name would be "vm_pager_page_dirtied()"
|
|
|
|
|
* XXX: It is not obvious if this could be profitably used by any
|
|
|
|
|
* XXX: pagers besides the swap_pager or if it should even be a
|
|
|
|
|
* XXX: generic pager_op in the first place.
|
1999-01-21 08:29:12 +00:00
|
|
|
*/
|
|
|
|
|
static __inline void
|
|
|
|
|
vm_pager_page_unswapped(vm_page_t m)
|
|
|
|
|
{
|
2021-05-01 00:28:39 +00:00
|
|
|
pgo_pageunswapped_t *method;
|
2003-10-24 06:43:04 +00:00
|
|
|
|
2021-05-01 00:28:39 +00:00
|
|
|
method = pagertab[m->object->type]->pgo_pageunswapped;
|
|
|
|
|
if (method != NULL)
|
|
|
|
|
method(m);
|
1999-01-21 08:29:12 +00:00
|
|
|
}
|
|
|
|
|
|
2019-09-03 20:31:48 +00:00
|
|
|
static __inline void
|
|
|
|
|
vm_pager_update_writecount(vm_object_t object, vm_offset_t start,
|
|
|
|
|
vm_offset_t end)
|
|
|
|
|
{
|
2021-05-01 00:28:39 +00:00
|
|
|
pgo_writecount_t *method;
|
2019-09-03 20:31:48 +00:00
|
|
|
|
2021-05-01 00:28:39 +00:00
|
|
|
method = pagertab[object->type]->pgo_update_writecount;
|
|
|
|
|
if (method != NULL)
|
|
|
|
|
method(object, start, end);
|
2019-09-03 20:31:48 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static __inline void
|
|
|
|
|
vm_pager_release_writecount(vm_object_t object, vm_offset_t start,
|
|
|
|
|
vm_offset_t end)
|
|
|
|
|
{
|
2021-05-01 00:28:39 +00:00
|
|
|
pgo_writecount_t *method;
|
2019-09-03 20:31:48 +00:00
|
|
|
|
2021-05-01 00:28:39 +00:00
|
|
|
method = pagertab[object->type]->pgo_release_writecount;
|
|
|
|
|
if (method != NULL)
|
|
|
|
|
method(object, start, end);
|
2019-09-03 20:31:48 +00:00
|
|
|
}
|
|
|
|
|
|
2021-05-01 01:04:04 +00:00
|
|
|
static __inline void
|
|
|
|
|
vm_pager_getvp(vm_object_t object, struct vnode **vpp, bool *vp_heldp)
|
|
|
|
|
{
|
|
|
|
|
pgo_getvp_t *method;
|
|
|
|
|
|
|
|
|
|
*vpp = NULL;
|
2021-05-01 01:08:28 +00:00
|
|
|
if (vp_heldp != NULL)
|
|
|
|
|
*vp_heldp = false;
|
2021-05-01 01:04:04 +00:00
|
|
|
method = pagertab[object->type]->pgo_getvp;
|
2021-05-01 01:08:28 +00:00
|
|
|
if (method != NULL)
|
2021-05-01 01:04:04 +00:00
|
|
|
method(object, vpp, vp_heldp);
|
|
|
|
|
}
|
|
|
|
|
|
2021-05-01 17:12:32 +00:00
|
|
|
static __inline void
|
|
|
|
|
vm_pager_freespace(vm_object_t object, vm_pindex_t start,
|
|
|
|
|
vm_size_t size)
|
|
|
|
|
{
|
|
|
|
|
pgo_freespace_t *method;
|
|
|
|
|
|
|
|
|
|
method = pagertab[object->type]->pgo_freespace;
|
|
|
|
|
if (method != NULL)
|
|
|
|
|
method(object, start, size);
|
|
|
|
|
}
|
|
|
|
|
|
2021-05-07 19:06:16 +00:00
|
|
|
int vm_pager_alloc_dyn_type(struct pagerops *ops, int base_type);
|
|
|
|
|
void vm_pager_free_dyn_type(objtype_t type);
|
|
|
|
|
|
2011-11-15 14:40:00 +00:00
|
|
|
struct cdev_pager_ops {
|
|
|
|
|
int (*cdev_pg_fault)(vm_object_t vm_obj, vm_ooffset_t offset,
|
|
|
|
|
int prot, vm_page_t *mres);
|
Add a new populate() pager method and extend device pager ops vector
with cdev_pg_populate() to provide device drivers access to it. It
gives drivers fine control of the pages ownership and allows drivers
to implement arbitrary prefault policies.
The populate method is called on a page fault and is supposed to
populate the vm object with the page at the fault location and some
amount of pages around it, at pager's discretion. VM provides the
pager with the hints about current range of the object mapping, to
avoid instantiation of immediately unused pages, if pager decides so.
Also, VM passes the fault type and map entry protection to the pager,
allowing it to force the optimal required ownership of the mapped
pages.
Installed pages must contiguously fill the returned region, be fully
valid and exclusively busied. Of course, the pages must be compatible
with the object' type.
After populate() successfully returned, VM fault handler installs as
many instantiated pages into the process page tables as it sees
reasonable, while still obeying the correct semantic for COW and vm
map locking.
The method is opt-in, pager sets OBJ_POPULATE flag to indicate that
the method can be called. If pager' vm objects can be shadowed, pager
must implement the traditional getpages() method in addition to the
populate(). Populate() might fall back to the getpages() on per-call
basis as well, by returning VM_PAGER_BAD error code.
For now for device pagers, the populate() method is only allowed to be
used by the managed device pagers, but the limitation is only made
because there is no unmanaged fault handlers which could use it right
now.
KPI designed together with, and reviewed by: alc
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
2016-12-08 11:26:11 +00:00
|
|
|
int (*cdev_pg_populate)(vm_object_t vm_obj, vm_pindex_t pidx,
|
|
|
|
|
int fault_type, vm_prot_t max_prot, vm_pindex_t *first,
|
|
|
|
|
vm_pindex_t *last);
|
2011-11-15 14:40:00 +00:00
|
|
|
int (*cdev_pg_ctor)(void *handle, vm_ooffset_t size, vm_prot_t prot,
|
|
|
|
|
vm_ooffset_t foff, struct ucred *cred, u_short *color);
|
|
|
|
|
void (*cdev_pg_dtor)(void *handle);
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
vm_object_t cdev_pager_allocate(void *handle, enum obj_type tp,
|
2021-05-03 21:58:07 +00:00
|
|
|
const struct cdev_pager_ops *ops, vm_ooffset_t size, vm_prot_t prot,
|
2011-11-15 14:40:00 +00:00
|
|
|
vm_ooffset_t foff, struct ucred *cred);
|
|
|
|
|
vm_object_t cdev_pager_lookup(void *handle);
|
|
|
|
|
void cdev_pager_free_page(vm_object_t object, vm_page_t m);
|
|
|
|
|
|
2020-09-09 00:00:43 +00:00
|
|
|
struct phys_pager_ops {
|
|
|
|
|
int (*phys_pg_getpages)(vm_object_t vm_obj, vm_page_t *m, int count,
|
|
|
|
|
int *rbehind, int *rahead);
|
|
|
|
|
int (*phys_pg_populate)(vm_object_t vm_obj, vm_pindex_t pidx,
|
|
|
|
|
int fault_type, vm_prot_t max_prot, vm_pindex_t *first,
|
|
|
|
|
vm_pindex_t *last);
|
|
|
|
|
boolean_t (*phys_pg_haspage)(vm_object_t obj, vm_pindex_t pindex,
|
|
|
|
|
int *before, int *after);
|
|
|
|
|
void (*phys_pg_ctor)(vm_object_t vm_obj, vm_prot_t prot,
|
|
|
|
|
vm_ooffset_t foff, struct ucred *cred);
|
|
|
|
|
void (*phys_pg_dtor)(vm_object_t vm_obj);
|
|
|
|
|
};
|
2021-05-03 21:58:07 +00:00
|
|
|
extern const struct phys_pager_ops default_phys_pg_ops;
|
|
|
|
|
vm_object_t phys_pager_allocate(void *handle, const struct phys_pager_ops *ops,
|
2020-09-09 00:00:43 +00:00
|
|
|
void *data, vm_ooffset_t size, vm_prot_t prot, vm_ooffset_t foff,
|
|
|
|
|
struct ucred *cred);
|
|
|
|
|
|
2002-03-10 21:52:48 +00:00
|
|
|
#endif /* _KERNEL */
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
#endif /* _VM_PAGER_ */
|
