2005-01-07 02:29:27 +00:00
|
|
|
/*-
|
1994-05-24 10:09:53 +00:00
|
|
|
* Copyright (c) 1990 University of Utah.
|
|
|
|
|
* Copyright (c) 1991, 1993
|
|
|
|
|
* The Regents of the University of California. All rights reserved.
|
|
|
|
|
*
|
|
|
|
|
* This code is derived from software contributed to Berkeley by
|
|
|
|
|
* the Systems Programming Group of the University of Utah Computer
|
|
|
|
|
* Science Department.
|
|
|
|
|
*
|
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
|
* 4. Neither the name of the University nor the names of its contributors
|
|
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
|
* without specific prior written permission.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
|
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
|
*
|
|
|
|
|
* @(#)vm_pager.h 8.4 (Berkeley) 1/12/94
|
1999-08-28 01:08:13 +00:00
|
|
|
* $FreeBSD$
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Pager routine interface definition.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef _VM_PAGER_
|
|
|
|
|
#define _VM_PAGER_
|
|
|
|
|
|
1998-02-03 22:19:35 +00:00
|
|
|
#include <sys/queue.h>
|
|
|
|
|
|
2000-05-26 02:09:24 +00:00
|
|
|
TAILQ_HEAD(pagerlst, vm_object);
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2004-11-09 13:43:20 +00:00
|
|
|
typedef void pgo_init_t(void);
|
Implement global and per-uid accounting of the anonymous memory. Add
rlimit RLIMIT_SWAP that limits the amount of swap that may be reserved
for the uid.
The accounting information (charge) is associated with either map entry,
or vm object backing the entry, assuming the object is the first one
in the shadow chain and entry does not require COW. Charge is moved
from entry to object on allocation of the object, e.g. during the mmap,
assuming the object is allocated, or on the first page fault on the
entry. It moves back to the entry on forks due to COW setup.
The per-entry granularity of accounting makes the charge process fair
for processes that change uid during lifetime, and decrements charge
for proper uid when region is unmapped.
The interface of vm_pager_allocate(9) is extended by adding struct ucred *,
that is used to charge appropriate uid when allocation if performed by
kernel, e.g. md(4).
Several syscalls, among them is fork(2), may now return ENOMEM when
global or per-uid limits are enforced.
In collaboration with: pho
Reviewed by: alc
Approved by: re (kensmith)
2009-06-23 20:45:22 +00:00
|
|
|
typedef vm_object_t pgo_alloc_t(void *, vm_ooffset_t, vm_prot_t, vm_ooffset_t,
|
|
|
|
|
struct ucred *);
|
2004-11-09 13:43:20 +00:00
|
|
|
typedef void pgo_dealloc_t(vm_object_t);
|
|
|
|
|
typedef int pgo_getpages_t(vm_object_t, vm_page_t *, int, int);
|
2014-11-23 12:01:52 +00:00
|
|
|
typedef void pgo_getpages_iodone_t(void *, vm_page_t *, int, int);
|
|
|
|
|
typedef int pgo_getpages_async_t(vm_object_t, vm_page_t *, int, int,
|
|
|
|
|
pgo_getpages_iodone_t, void *);
|
2004-11-09 13:43:20 +00:00
|
|
|
typedef void pgo_putpages_t(vm_object_t, vm_page_t *, int, int, int *);
|
|
|
|
|
typedef boolean_t pgo_haspage_t(vm_object_t, vm_pindex_t, int *, int *);
|
|
|
|
|
typedef void pgo_pageunswapped_t(vm_page_t);
|
|
|
|
|
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
struct pagerops {
|
2014-11-14 18:15:35 +00:00
|
|
|
pgo_init_t *pgo_init; /* Initialize pager. */
|
|
|
|
|
pgo_alloc_t *pgo_alloc; /* Allocate pager. */
|
|
|
|
|
pgo_dealloc_t *pgo_dealloc; /* Disassociate. */
|
|
|
|
|
pgo_getpages_t *pgo_getpages; /* Get (read) page. */
|
2014-11-23 12:01:52 +00:00
|
|
|
pgo_getpages_async_t *pgo_getpages_async; /* Get page asyncly. */
|
2014-11-14 18:15:35 +00:00
|
|
|
pgo_putpages_t *pgo_putpages; /* Put (write) page. */
|
|
|
|
|
pgo_haspage_t *pgo_haspage; /* Query page. */
|
2014-11-14 18:00:00 +00:00
|
|
|
pgo_pageunswapped_t *pgo_pageunswapped;
|
1994-05-24 10:09:53 +00:00
|
|
|
};
|
|
|
|
|
|
2003-08-03 09:27:39 +00:00
|
|
|
extern struct pagerops defaultpagerops;
|
|
|
|
|
extern struct pagerops swappagerops;
|
|
|
|
|
extern struct pagerops vnodepagerops;
|
|
|
|
|
extern struct pagerops devicepagerops;
|
|
|
|
|
extern struct pagerops physpagerops;
|
2009-07-24 13:50:29 +00:00
|
|
|
extern struct pagerops sgpagerops;
|
2012-05-12 20:49:58 +00:00
|
|
|
extern struct pagerops mgtdevicepagerops;
|
2003-08-03 09:27:39 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
|
* get/put return values
|
|
|
|
|
* OK operation was successful
|
|
|
|
|
* BAD specified data was out of the accepted range
|
|
|
|
|
* FAIL specified data was in range, but doesn't exist
|
|
|
|
|
* PEND operations was initiated but not completed
|
|
|
|
|
* ERROR error while accessing data that is in range and exists
|
|
|
|
|
* AGAIN temporary resource shortage prevented operation from happening
|
|
|
|
|
*/
|
|
|
|
|
#define VM_PAGER_OK 0
|
|
|
|
|
#define VM_PAGER_BAD 1
|
|
|
|
|
#define VM_PAGER_FAIL 2
|
|
|
|
|
#define VM_PAGER_PEND 3
|
|
|
|
|
#define VM_PAGER_ERROR 4
|
|
|
|
|
#define VM_PAGER_AGAIN 5
|
|
|
|
|
|
2002-12-28 21:03:42 +00:00
|
|
|
#define VM_PAGER_PUT_SYNC 0x0001
|
|
|
|
|
#define VM_PAGER_PUT_INVAL 0x0002
|
|
|
|
|
#define VM_PAGER_CLUSTER_OK 0x0008
|
This mega-commit is meant to fix numerous interrelated problems. There
has been some bitrot and incorrect assumptions in the vfs_bio code. These
problems have manifest themselves worse on NFS type filesystems, but can
still affect local filesystems under certain circumstances. Most of
the problems have involved mmap consistancy, and as a side-effect broke
the vfs.ioopt code. This code might have been committed seperately, but
almost everything is interrelated.
1) Allow (pmap_object_init_pt) prefaulting of buffer-busy pages that
are fully valid.
2) Rather than deactivating erroneously read initial (header) pages in
kern_exec, we now free them.
3) Fix the rundown of non-VMIO buffers that are in an inconsistent
(missing vp) state.
4) Fix the disassociation of pages from buffers in brelse. The previous
code had rotted and was faulty in a couple of important circumstances.
5) Remove a gratuitious buffer wakeup in vfs_vmio_release.
6) Remove a crufty and currently unused cluster mechanism for VBLK
files in vfs_bio_awrite. When the code is functional, I'll add back
a cleaner version.
7) The page busy count wakeups assocated with the buffer cache usage were
incorrectly cleaned up in a previous commit by me. Revert to the
original, correct version, but with a cleaner implementation.
8) The cluster read code now tries to keep data associated with buffers
more aggressively (without breaking the heuristics) when it is presumed
that the read data (buffers) will be soon needed.
9) Change to filesystem lockmgr locks so that they use LK_NOPAUSE. The
delay loop waiting is not useful for filesystem locks, due to the
length of the time intervals.
10) Correct and clean-up spec_getpages.
11) Implement a fully functional nfs_getpages, nfs_putpages.
12) Fix nfs_write so that modifications are coherent with the NFS data on
the server disk (at least as well as NFS seems to allow.)
13) Properly support MS_INVALIDATE on NFS.
14) Properly pass down MS_INVALIDATE to lower levels of the VM code from
vm_map_clean.
15) Better support the notion of pages being busy but valid, so that
fewer in-transit waits occur. (use p->busy more for pageouts instead
of PG_BUSY.) Since the page is fully valid, it is still usable for
reads.
16) It is possible (in error) for cached pages to be busy. Make the
page allocation code handle that case correctly. (It should probably
be a printf or panic, but I want the system to handle coding errors
robustly. I'll probably add a printf.)
17) Correct the design and usage of vm_page_sleep. It didn't handle
consistancy problems very well, so make the design a little less
lofty. After vm_page_sleep, if it ever blocked, it is still important
to relookup the page (if the object generation count changed), and
verify it's status (always.)
18) In vm_pageout.c, vm_pageout_clean had rotted, so clean that up.
19) Push the page busy for writes and VM_PROT_READ into vm_pageout_flush.
20) Fix vm_pager_put_pages and it's descendents to support an int flag
instead of a boolean, so that we can pass down the invalidate bit.
1998-03-07 21:37:31 +00:00
|
|
|
|
1999-12-29 05:07:58 +00:00
|
|
|
#ifdef _KERNEL
|
1997-10-12 20:26:33 +00:00
|
|
|
|
1999-01-21 08:29:12 +00:00
|
|
|
extern struct pagerops *pagertab[];
|
2013-06-28 03:51:20 +00:00
|
|
|
extern struct mtx_padalign pbuf_mtx;
|
1995-07-29 11:44:31 +00:00
|
|
|
|
Implement global and per-uid accounting of the anonymous memory. Add
rlimit RLIMIT_SWAP that limits the amount of swap that may be reserved
for the uid.
The accounting information (charge) is associated with either map entry,
or vm object backing the entry, assuming the object is the first one
in the shadow chain and entry does not require COW. Charge is moved
from entry to object on allocation of the object, e.g. during the mmap,
assuming the object is allocated, or on the first page fault on the
entry. It moves back to the entry on forks due to COW setup.
The per-entry granularity of accounting makes the charge process fair
for processes that change uid during lifetime, and decrements charge
for proper uid when region is unmapped.
The interface of vm_pager_allocate(9) is extended by adding struct ucred *,
that is used to charge appropriate uid when allocation if performed by
kernel, e.g. md(4).
Several syscalls, among them is fork(2), may now return ENOMEM when
global or per-uid limits are enforced.
In collaboration with: pho
Reviewed by: alc
Approved by: re (kensmith)
2009-06-23 20:45:22 +00:00
|
|
|
vm_object_t vm_pager_allocate(objtype_t, void *, vm_ooffset_t, vm_prot_t,
|
|
|
|
|
vm_ooffset_t, struct ucred *);
|
2002-03-19 22:20:14 +00:00
|
|
|
void vm_pager_bufferinit(void);
|
|
|
|
|
void vm_pager_deallocate(vm_object_t);
|
2015-06-17 22:44:27 +00:00
|
|
|
int vm_pager_get_pages(vm_object_t, vm_page_t *, int, int);
|
|
|
|
|
int vm_pager_get_pages_async(vm_object_t, vm_page_t *, int, int,
|
|
|
|
|
pgo_getpages_iodone_t, void *);
|
2002-03-19 22:20:14 +00:00
|
|
|
static __inline boolean_t vm_pager_has_page(vm_object_t, vm_pindex_t, int *, int *);
|
|
|
|
|
void vm_pager_init(void);
|
|
|
|
|
vm_object_t vm_pager_object_lookup(struct pagerlst *, void *);
|
2014-09-14 18:07:55 +00:00
|
|
|
void vm_pager_free_nonreq(vm_object_t object, vm_page_t ma[], int reqpage,
|
2015-03-17 19:19:19 +00:00
|
|
|
int npages, boolean_t object_locked);
|
1999-01-21 08:29:12 +00:00
|
|
|
|
1999-01-24 02:32:15 +00:00
|
|
|
static __inline void
|
1999-01-21 08:29:12 +00:00
|
|
|
vm_pager_put_pages(
|
|
|
|
|
vm_object_t object,
|
|
|
|
|
vm_page_t *m,
|
|
|
|
|
int count,
|
|
|
|
|
int flags,
|
|
|
|
|
int *rtvals
|
|
|
|
|
) {
|
2003-10-24 06:43:04 +00:00
|
|
|
|
2013-02-21 21:54:53 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
1999-01-24 02:32:15 +00:00
|
|
|
(*pagertab[object->type]->pgo_putpages)
|
|
|
|
|
(object, m, count, flags, rtvals);
|
1999-01-21 08:29:12 +00:00
|
|
|
}
|
|
|
|
|
|
2000-03-27 21:33:32 +00:00
|
|
|
/*
|
|
|
|
|
* vm_pager_haspage
|
|
|
|
|
*
|
|
|
|
|
* Check to see if an object's pager has the requested page. The
|
|
|
|
|
* object's pager will also set before and after to give the caller
|
|
|
|
|
* some idea of the number of pages before and after the requested
|
|
|
|
|
* page can be I/O'd efficiently.
|
|
|
|
|
*
|
2005-05-18 22:08:52 +00:00
|
|
|
* The object must be locked.
|
2000-03-27 21:33:32 +00:00
|
|
|
*/
|
1999-01-21 08:29:12 +00:00
|
|
|
static __inline boolean_t
|
|
|
|
|
vm_pager_has_page(
|
|
|
|
|
vm_object_t object,
|
|
|
|
|
vm_pindex_t offset,
|
|
|
|
|
int *before,
|
|
|
|
|
int *after
|
|
|
|
|
) {
|
2001-05-19 01:28:09 +00:00
|
|
|
boolean_t ret;
|
|
|
|
|
|
2013-02-21 21:54:53 +00:00
|
|
|
VM_OBJECT_ASSERT_WLOCKED(object);
|
2001-05-19 01:28:09 +00:00
|
|
|
ret = (*pagertab[object->type]->pgo_haspage)
|
|
|
|
|
(object, offset, before, after);
|
|
|
|
|
return (ret);
|
1999-01-21 08:29:12 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* vm_pager_page_unswapped
|
|
|
|
|
*
|
2005-05-18 22:08:52 +00:00
|
|
|
* Destroy swap associated with the page.
|
1999-01-21 08:29:12 +00:00
|
|
|
*
|
2005-05-18 22:08:52 +00:00
|
|
|
* The object containing the page must be locked.
|
1999-01-21 08:29:12 +00:00
|
|
|
* This function may not block.
|
2003-08-06 10:51:40 +00:00
|
|
|
*
|
|
|
|
|
* XXX: A much better name would be "vm_pager_page_dirtied()"
|
|
|
|
|
* XXX: It is not obvious if this could be profitably used by any
|
|
|
|
|
* XXX: pagers besides the swap_pager or if it should even be a
|
|
|
|
|
* XXX: generic pager_op in the first place.
|
1999-01-21 08:29:12 +00:00
|
|
|
*/
|
|
|
|
|
static __inline void
|
|
|
|
|
vm_pager_page_unswapped(vm_page_t m)
|
|
|
|
|
{
|
2003-10-24 06:43:04 +00:00
|
|
|
|
2014-08-06 19:34:03 +00:00
|
|
|
VM_OBJECT_ASSERT_LOCKED(m->object);
|
1999-01-21 08:29:12 +00:00
|
|
|
if (pagertab[m->object->type]->pgo_pageunswapped)
|
|
|
|
|
(*pagertab[m->object->type]->pgo_pageunswapped)(m);
|
|
|
|
|
}
|
|
|
|
|
|
2011-11-15 14:40:00 +00:00
|
|
|
struct cdev_pager_ops {
|
|
|
|
|
int (*cdev_pg_fault)(vm_object_t vm_obj, vm_ooffset_t offset,
|
|
|
|
|
int prot, vm_page_t *mres);
|
|
|
|
|
int (*cdev_pg_ctor)(void *handle, vm_ooffset_t size, vm_prot_t prot,
|
|
|
|
|
vm_ooffset_t foff, struct ucred *cred, u_short *color);
|
|
|
|
|
void (*cdev_pg_dtor)(void *handle);
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
vm_object_t cdev_pager_allocate(void *handle, enum obj_type tp,
|
|
|
|
|
struct cdev_pager_ops *ops, vm_ooffset_t size, vm_prot_t prot,
|
|
|
|
|
vm_ooffset_t foff, struct ucred *cred);
|
|
|
|
|
vm_object_t cdev_pager_lookup(void *handle);
|
|
|
|
|
void cdev_pager_free_page(vm_object_t object, vm_page_t m);
|
|
|
|
|
|
2002-03-10 21:52:48 +00:00
|
|
|
#endif /* _KERNEL */
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
#endif /* _VM_PAGER_ */
|
