These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
/*
|
1994-05-25 09:21:21 +00:00
|
|
|
* Copyright (c) 1991 Regents of the University of California.
|
|
|
|
* All rights reserved.
|
1994-05-24 10:09:53 +00:00
|
|
|
*
|
|
|
|
* This code is derived from software contributed to Berkeley by
|
|
|
|
* The Mach Operating System project at Carnegie-Mellon University.
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
* 3. All advertising materials mentioning features or use of this software
|
2000-03-27 20:41:17 +00:00
|
|
|
* must display the following acknowledgement:
|
1994-05-24 10:09:53 +00:00
|
|
|
* This product includes software developed by the University of
|
|
|
|
* California, Berkeley and its contributors.
|
|
|
|
* 4. Neither the name of the University nor the names of its contributors
|
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
* without specific prior written permission.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
*
|
1994-05-25 09:21:21 +00:00
|
|
|
* from: @(#)vm_page.c 7.4 (Berkeley) 5/7/91
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Copyright (c) 1987, 1990 Carnegie-Mellon University.
|
|
|
|
* All rights reserved.
|
|
|
|
*
|
|
|
|
* Authors: Avadis Tevanian, Jr., Michael Wayne Young
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Permission to use, copy, modify and distribute this software and
|
|
|
|
* its documentation is hereby granted, provided that both the copyright
|
|
|
|
* notice and this permission notice appear in all copies of the
|
|
|
|
* software, derivative works or modified versions, and any portions
|
|
|
|
* thereof, and that both notices appear in supporting documentation.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
|
|
|
* CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS"
|
|
|
|
* CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND
|
1994-05-24 10:09:53 +00:00
|
|
|
* FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Carnegie Mellon requests users of this software to return to
|
|
|
|
*
|
|
|
|
* Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU
|
|
|
|
* School of Computer Science
|
|
|
|
* Carnegie Mellon University
|
|
|
|
* Pittsburgh PA 15213-3890
|
|
|
|
*
|
|
|
|
* any improvements or extensions that they make and grant Carnegie the
|
|
|
|
* rights to redistribute these changes.
|
|
|
|
*/
|
|
|
|
|
2001-07-04 23:27:09 +00:00
|
|
|
/*
|
|
|
|
* GENERAL RULES ON VM_PAGE MANIPULATION
|
|
|
|
*
|
|
|
|
* - a pageq mutex is required when adding or removing a page from a
|
|
|
|
* page queue (vm_page_queue[]), regardless of other mutexes or the
|
|
|
|
* busy state of a page.
|
|
|
|
*
|
|
|
|
* - a hash chain mutex is required when associating or disassociating
|
|
|
|
* a page from the VM PAGE CACHE hash table (vm_page_buckets),
|
|
|
|
* regardless of other mutexes or the busy state of a page.
|
|
|
|
*
|
|
|
|
* - either a hash chain mutex OR a busied page is required in order
|
|
|
|
* to modify the page flags. A hash chain mutex must be obtained in
|
|
|
|
* order to busy a page. A page's flags cannot be modified by a
|
|
|
|
* hash chain mutex if the page is marked busy.
|
|
|
|
*
|
|
|
|
* - The object memq mutex is held when inserting or removing
|
|
|
|
* pages from an object (vm_page_insert() or vm_page_remove()). This
|
|
|
|
* is different from the object's main mutex.
|
|
|
|
*
|
|
|
|
* Generally speaking, you have to be aware of side effects when running
|
|
|
|
* vm_page ops. A vm_page_lookup() will return with the hash chain
|
|
|
|
* locked, whether it was able to lookup the page or not. vm_page_free(),
|
|
|
|
* vm_page_cache(), vm_page_activate(), and a number of other routines
|
|
|
|
* will release the hash chain mutex for you. Intermediate manipulation
|
|
|
|
* routines such as vm_page_flag_set() expect the hash chain to be held
|
|
|
|
* on entry and the hash chain will remain held on return.
|
|
|
|
*
|
|
|
|
* pageq scanning can only occur with the pageq in question locked.
|
|
|
|
* We have a known bottleneck with the active queue, but the cache
|
|
|
|
* and free queues are actually arrays already.
|
|
|
|
*/
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* Resident memory management module.
|
|
|
|
*/
|
|
|
|
|
2003-06-11 23:50:51 +00:00
|
|
|
#include <sys/cdefs.h>
|
|
|
|
__FBSDID("$FreeBSD$");
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/param.h>
|
|
|
|
#include <sys/systm.h>
|
2001-05-01 08:13:21 +00:00
|
|
|
#include <sys/lock.h>
|
1996-01-27 00:13:33 +00:00
|
|
|
#include <sys/malloc.h>
|
2001-05-22 07:01:11 +00:00
|
|
|
#include <sys/mutex.h>
|
1994-05-25 09:21:21 +00:00
|
|
|
#include <sys/proc.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <sys/vmmeter.h>
|
1997-12-29 00:25:11 +00:00
|
|
|
#include <sys/vnode.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
#include <vm/vm.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_param.h>
|
1995-03-16 18:17:34 +00:00
|
|
|
#include <vm/vm_kern.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_object.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm_page.h>
|
|
|
|
#include <vm/vm_pageout.h>
|
1999-01-21 08:29:12 +00:00
|
|
|
#include <vm/vm_pager.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_extern.h>
|
2002-03-19 09:11:49 +00:00
|
|
|
#include <vm/uma.h>
|
|
|
|
#include <vm/uma_int.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Associated with page of user-allocatable memory is a
|
|
|
|
* page structure.
|
|
|
|
*/
|
1996-01-19 04:00:31 +00:00
|
|
|
|
2002-07-04 22:07:37 +00:00
|
|
|
struct mtx vm_page_queue_mtx;
|
|
|
|
struct mtx vm_page_queue_free_mtx;
|
|
|
|
|
1996-11-17 02:38:31 +00:00
|
|
|
vm_page_t vm_page_array = 0;
|
2000-05-21 12:50:18 +00:00
|
|
|
int vm_page_array_size = 0;
|
1996-11-17 02:38:31 +00:00
|
|
|
long first_page = 0;
|
|
|
|
int vm_page_zero_count = 0;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* vm_set_page_size:
|
|
|
|
*
|
|
|
|
* Sets the page size, perhaps based upon the memory
|
|
|
|
* size. Must be called before any use of page-size
|
|
|
|
* dependent functions.
|
|
|
|
*/
|
1995-05-30 08:16:23 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_set_page_size(void)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
|
|
|
if (cnt.v_page_size == 0)
|
1999-06-20 04:55:29 +00:00
|
|
|
cnt.v_page_size = PAGE_SIZE;
|
1999-06-22 07:18:20 +00:00
|
|
|
if (((cnt.v_page_size - 1) & cnt.v_page_size) != 0)
|
1994-05-24 10:09:53 +00:00
|
|
|
panic("vm_set_page_size: page size not a power of two");
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_startup:
|
|
|
|
*
|
|
|
|
* Initializes the resident memory module.
|
|
|
|
*
|
|
|
|
* Allocates memory for the page cells, and
|
|
|
|
* for the object/offset-to-page hash table headers.
|
|
|
|
* Each page cell is initialized and placed on the free list.
|
|
|
|
*/
|
1994-05-25 09:21:21 +00:00
|
|
|
vm_offset_t
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_startup(vm_offset_t starta, vm_offset_t enda, vm_offset_t vaddr)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2001-07-04 19:00:13 +00:00
|
|
|
vm_offset_t mapped;
|
2003-03-25 00:07:06 +00:00
|
|
|
vm_size_t npages;
|
|
|
|
vm_paddr_t page_range;
|
|
|
|
vm_paddr_t new_end;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
int i;
|
2003-03-25 00:07:06 +00:00
|
|
|
vm_paddr_t pa;
|
1994-05-25 09:21:21 +00:00
|
|
|
int nblocks;
|
2003-03-25 00:07:06 +00:00
|
|
|
vm_paddr_t last_pa;
|
1994-05-25 09:21:21 +00:00
|
|
|
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
/* the biggest memory array is the second group of pages */
|
2003-03-25 00:07:06 +00:00
|
|
|
vm_paddr_t end;
|
|
|
|
vm_paddr_t biggestsize;
|
|
|
|
int biggestone;
|
1994-05-25 09:21:21 +00:00
|
|
|
|
2003-03-25 00:07:06 +00:00
|
|
|
vm_paddr_t total;
|
2002-03-19 09:11:49 +00:00
|
|
|
vm_size_t bootpages;
|
1994-05-25 09:21:21 +00:00
|
|
|
|
|
|
|
total = 0;
|
|
|
|
biggestsize = 0;
|
|
|
|
biggestone = 0;
|
|
|
|
nblocks = 0;
|
|
|
|
vaddr = round_page(vaddr);
|
|
|
|
|
|
|
|
for (i = 0; phys_avail[i + 1]; i += 2) {
|
|
|
|
phys_avail[i] = round_page(phys_avail[i]);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
phys_avail[i + 1] = trunc_page(phys_avail[i + 1]);
|
1994-05-25 09:21:21 +00:00
|
|
|
}
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
1994-05-25 09:21:21 +00:00
|
|
|
for (i = 0; phys_avail[i + 1]; i += 2) {
|
2003-03-25 00:07:06 +00:00
|
|
|
vm_paddr_t size = phys_avail[i + 1] - phys_avail[i];
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
1994-05-25 09:21:21 +00:00
|
|
|
if (size > biggestsize) {
|
|
|
|
biggestone = i;
|
|
|
|
biggestsize = size;
|
|
|
|
}
|
|
|
|
++nblocks;
|
|
|
|
total += size;
|
|
|
|
}
|
|
|
|
|
2001-03-01 19:21:24 +00:00
|
|
|
end = phys_avail[biggestone+1];
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2002-07-04 22:07:37 +00:00
|
|
|
/*
|
|
|
|
* Initialize the locks.
|
|
|
|
*/
|
|
|
|
mtx_init(&vm_page_queue_mtx, "vm page queue mutex", NULL, MTX_DEF);
|
|
|
|
mtx_init(&vm_page_queue_free_mtx, "vm page queue free mutex", NULL,
|
|
|
|
MTX_SPIN);
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
* Initialize the queue headers for the free queue, the active queue
|
|
|
|
* and the inactive queue.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2001-07-04 23:27:09 +00:00
|
|
|
vm_pageq_init();
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2002-03-19 09:11:49 +00:00
|
|
|
/*
|
2002-11-03 22:20:42 +00:00
|
|
|
* Allocate memory for use when boot strapping the kernel memory
|
|
|
|
* allocator.
|
2002-03-19 09:11:49 +00:00
|
|
|
*/
|
|
|
|
bootpages = UMA_BOOT_PAGES * UMA_SLAB_SIZE;
|
|
|
|
new_end = end - bootpages;
|
|
|
|
new_end = trunc_page(new_end);
|
|
|
|
mapped = pmap_map(&vaddr, new_end, end,
|
|
|
|
VM_PROT_READ | VM_PROT_WRITE);
|
|
|
|
bzero((caddr_t) mapped, end - new_end);
|
|
|
|
uma_startup((caddr_t)mapped);
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
* Compute the number of pages of memory that will be available for
|
|
|
|
* use (taking into account the overhead of a page structure per
|
|
|
|
* page).
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1994-05-25 09:21:21 +00:00
|
|
|
first_page = phys_avail[0] / PAGE_SIZE;
|
1999-06-22 07:18:20 +00:00
|
|
|
page_range = phys_avail[(nblocks - 1) * 2 + 1] / PAGE_SIZE - first_page;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
npages = (total - (page_range * sizeof(struct vm_page)) -
|
2001-03-01 19:21:24 +00:00
|
|
|
(end - new_end)) / PAGE_SIZE;
|
|
|
|
end = new_end;
|
2001-03-07 05:29:21 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
* Initialize the mem entry structures now, and put them in the free
|
|
|
|
* queue.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2001-03-01 19:21:24 +00:00
|
|
|
new_end = trunc_page(end - page_range * sizeof(struct vm_page));
|
2001-03-07 05:29:21 +00:00
|
|
|
mapped = pmap_map(&vaddr, new_end, end,
|
2001-03-01 19:21:24 +00:00
|
|
|
VM_PROT_READ | VM_PROT_WRITE);
|
2001-03-07 05:29:21 +00:00
|
|
|
vm_page_array = (vm_page_t) mapped;
|
2003-03-17 03:16:00 +00:00
|
|
|
phys_avail[biggestone + 1] = new_end;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
* Clear all of the page structures
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
bzero((caddr_t) vm_page_array, page_range * sizeof(struct vm_page));
|
|
|
|
vm_page_array_size = page_range;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
1999-03-19 05:21:03 +00:00
|
|
|
/*
|
|
|
|
* Construct the free queue(s) in descending order (by physical
|
|
|
|
* address) so that the first 16MB of physical memory is allocated
|
|
|
|
* last rather than first. On large-memory machines, this avoids
|
|
|
|
* the exhaustion of low physical memory before isa_dmainit has run.
|
|
|
|
*/
|
1994-05-25 09:21:21 +00:00
|
|
|
cnt.v_page_count = 0;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
cnt.v_free_count = 0;
|
1994-05-25 09:21:21 +00:00
|
|
|
for (i = 0; phys_avail[i + 1] && npages > 0; i += 2) {
|
2001-03-01 19:21:24 +00:00
|
|
|
pa = phys_avail[i];
|
2003-03-17 03:16:00 +00:00
|
|
|
last_pa = phys_avail[i + 1];
|
2001-03-01 19:21:24 +00:00
|
|
|
while (pa < last_pa && npages-- > 0) {
|
2001-07-04 23:27:09 +00:00
|
|
|
vm_pageq_add_new_page(pa);
|
1994-05-25 09:21:21 +00:00
|
|
|
pa += PAGE_SIZE;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
}
|
2001-03-07 05:29:21 +00:00
|
|
|
return (vaddr);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
2001-07-04 20:15:18 +00:00
|
|
|
void
|
|
|
|
vm_page_flag_set(vm_page_t m, unsigned short bits)
|
|
|
|
{
|
2002-12-19 07:23:46 +00:00
|
|
|
|
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
2001-07-31 04:09:52 +00:00
|
|
|
m->flags |= bits;
|
2001-07-04 20:15:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_flag_clear(vm_page_t m, unsigned short bits)
|
|
|
|
{
|
2002-12-28 22:49:37 +00:00
|
|
|
|
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
2001-07-31 04:09:52 +00:00
|
|
|
m->flags &= ~bits;
|
2001-07-04 20:15:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_busy(vm_page_t m)
|
|
|
|
{
|
|
|
|
KASSERT((m->flags & PG_BUSY) == 0,
|
|
|
|
("vm_page_busy: page already busy!!!"));
|
|
|
|
vm_page_flag_set(m, PG_BUSY);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_flash:
|
|
|
|
*
|
|
|
|
* wakeup anyone waiting for the page.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_flash(vm_page_t m)
|
|
|
|
{
|
|
|
|
if (m->flags & PG_WANTED) {
|
|
|
|
vm_page_flag_clear(m, PG_WANTED);
|
|
|
|
wakeup(m);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_wakeup:
|
|
|
|
*
|
|
|
|
* clear the PG_BUSY flag and wakeup anyone waiting for the
|
|
|
|
* page.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_wakeup(vm_page_t m)
|
|
|
|
{
|
|
|
|
KASSERT(m->flags & PG_BUSY, ("vm_page_wakeup: page not busy!!!"));
|
|
|
|
vm_page_flag_clear(m, PG_BUSY);
|
|
|
|
vm_page_flash(m);
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_io_start(vm_page_t m)
|
|
|
|
{
|
2002-07-31 07:27:08 +00:00
|
|
|
|
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
2001-07-31 04:09:52 +00:00
|
|
|
m->busy++;
|
2001-07-04 20:15:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_io_finish(vm_page_t m)
|
|
|
|
{
|
2002-08-01 17:57:42 +00:00
|
|
|
|
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
2001-07-31 04:09:52 +00:00
|
|
|
m->busy--;
|
2001-07-04 20:15:18 +00:00
|
|
|
if (m->busy == 0)
|
|
|
|
vm_page_flash(m);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Keep page from being freed by the page daemon
|
|
|
|
* much of the same effect as wiring, except much lower
|
|
|
|
* overhead and should be used only for *very* temporary
|
|
|
|
* holding ("wiring").
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_hold(vm_page_t mem)
|
|
|
|
{
|
2003-01-20 09:24:03 +00:00
|
|
|
|
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
2001-07-04 20:15:18 +00:00
|
|
|
mem->hold_count++;
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_unhold(vm_page_t mem)
|
|
|
|
{
|
2002-12-15 00:06:02 +00:00
|
|
|
|
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
2001-07-04 20:15:18 +00:00
|
|
|
--mem->hold_count;
|
|
|
|
KASSERT(mem->hold_count >= 0, ("vm_page_unhold: hold count < 0!!!"));
|
2002-02-19 23:19:30 +00:00
|
|
|
if (mem->hold_count == 0 && mem->queue == PQ_HOLD)
|
|
|
|
vm_page_free_toq(mem);
|
2001-07-04 20:15:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_copy:
|
|
|
|
*
|
|
|
|
* Copy one page to another
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_copy(vm_page_t src_m, vm_page_t dest_m)
|
|
|
|
{
|
2002-04-15 16:00:03 +00:00
|
|
|
pmap_copy_page(src_m, dest_m);
|
2001-07-04 20:15:18 +00:00
|
|
|
dest_m->valid = VM_PAGE_BITS_ALL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_free:
|
|
|
|
*
|
|
|
|
* Free a page
|
|
|
|
*
|
|
|
|
* The clearing of PG_ZERO is a temporary safety until the code can be
|
|
|
|
* reviewed to determine that PG_ZERO is being properly cleared on
|
|
|
|
* write faults or maps. PG_ZERO was previously cleared in
|
|
|
|
* vm_page_alloc().
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_free(vm_page_t m)
|
|
|
|
{
|
|
|
|
vm_page_flag_clear(m, PG_ZERO);
|
|
|
|
vm_page_free_toq(m);
|
2001-08-25 05:00:44 +00:00
|
|
|
vm_page_zero_idle_wakeup();
|
2001-07-04 20:15:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_free_zero:
|
|
|
|
*
|
|
|
|
* Free a page to the zerod-pages queue
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_free_zero(vm_page_t m)
|
|
|
|
{
|
|
|
|
vm_page_flag_set(m, PG_ZERO);
|
|
|
|
vm_page_free_toq(m);
|
|
|
|
}
|
|
|
|
|
2002-07-29 19:41:22 +00:00
|
|
|
/*
|
|
|
|
* vm_page_sleep_if_busy:
|
|
|
|
*
|
|
|
|
* Sleep and release the page queues lock if PG_BUSY is set or,
|
|
|
|
* if also_m_busy is TRUE, busy is non-zero. Returns TRUE if the
|
|
|
|
* thread slept and the page queues lock was released.
|
|
|
|
* Otherwise, retains the page queues lock and returns FALSE.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
vm_page_sleep_if_busy(vm_page_t m, int also_m_busy, const char *msg)
|
|
|
|
{
|
2003-03-01 19:16:32 +00:00
|
|
|
int is_object_locked;
|
2002-07-29 19:41:22 +00:00
|
|
|
|
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
|
|
|
if ((m->flags & PG_BUSY) || (also_m_busy && m->busy)) {
|
|
|
|
vm_page_flag_set(m, PG_WANTED | PG_REFERENCED);
|
2003-03-01 19:16:32 +00:00
|
|
|
/*
|
|
|
|
* Remove mtx_owned() after vm_object locking is finished.
|
|
|
|
*/
|
|
|
|
if ((is_object_locked = m->object != NULL &&
|
|
|
|
mtx_owned(&m->object->mtx)))
|
|
|
|
mtx_unlock(&m->object->mtx);
|
2002-07-29 19:41:22 +00:00
|
|
|
msleep(m, &vm_page_queue_mtx, PDROP | PVM, msg, 0);
|
2003-03-01 19:16:32 +00:00
|
|
|
if (is_object_locked)
|
|
|
|
mtx_lock(&m->object->mtx);
|
2002-07-29 19:41:22 +00:00
|
|
|
return (TRUE);
|
|
|
|
}
|
|
|
|
return (FALSE);
|
|
|
|
}
|
|
|
|
|
2001-07-04 20:15:18 +00:00
|
|
|
/*
|
|
|
|
* vm_page_dirty:
|
|
|
|
*
|
|
|
|
* make page all dirty
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_dirty(vm_page_t m)
|
|
|
|
{
|
|
|
|
KASSERT(m->queue - m->pc != PQ_CACHE,
|
|
|
|
("vm_page_dirty: page in cache!"));
|
2003-02-24 17:30:45 +00:00
|
|
|
KASSERT(m->queue - m->pc != PQ_FREE,
|
|
|
|
("vm_page_dirty: page is free!"));
|
2001-07-04 20:15:18 +00:00
|
|
|
m->dirty = VM_PAGE_BITS_ALL;
|
|
|
|
}
|
|
|
|
|
2002-10-18 17:24:30 +00:00
|
|
|
/*
|
|
|
|
* vm_page_splay:
|
|
|
|
*
|
|
|
|
* Implements Sleator and Tarjan's top-down splay algorithm. Returns
|
|
|
|
* the vm_page containing the given pindex. If, however, that
|
|
|
|
* pindex is not found in the vm_object, returns a vm_page that is
|
|
|
|
* adjacent to the pindex, coming before or after it.
|
|
|
|
*/
|
2002-11-04 19:21:39 +00:00
|
|
|
vm_page_t
|
2002-10-18 17:24:30 +00:00
|
|
|
vm_page_splay(vm_pindex_t pindex, vm_page_t root)
|
|
|
|
{
|
|
|
|
struct vm_page dummy;
|
|
|
|
vm_page_t lefttreemax, righttreemin, y;
|
|
|
|
|
|
|
|
if (root == NULL)
|
|
|
|
return (root);
|
|
|
|
lefttreemax = righttreemin = &dummy;
|
|
|
|
for (;; root = y) {
|
|
|
|
if (pindex < root->pindex) {
|
|
|
|
if ((y = root->left) == NULL)
|
|
|
|
break;
|
|
|
|
if (pindex < y->pindex) {
|
|
|
|
/* Rotate right. */
|
|
|
|
root->left = y->right;
|
|
|
|
y->right = root;
|
|
|
|
root = y;
|
|
|
|
if ((y = root->left) == NULL)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
/* Link into the new root's right tree. */
|
|
|
|
righttreemin->left = root;
|
|
|
|
righttreemin = root;
|
|
|
|
} else if (pindex > root->pindex) {
|
|
|
|
if ((y = root->right) == NULL)
|
|
|
|
break;
|
|
|
|
if (pindex > y->pindex) {
|
|
|
|
/* Rotate left. */
|
|
|
|
root->right = y->left;
|
|
|
|
y->left = root;
|
|
|
|
root = y;
|
|
|
|
if ((y = root->right) == NULL)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
/* Link into the new root's left tree. */
|
|
|
|
lefttreemax->right = root;
|
|
|
|
lefttreemax = root;
|
|
|
|
} else
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
/* Assemble the new root. */
|
|
|
|
lefttreemax->right = root->left;
|
|
|
|
righttreemin->left = root->right;
|
|
|
|
root->left = dummy.right;
|
|
|
|
root->right = dummy.left;
|
|
|
|
return (root);
|
|
|
|
}
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* vm_page_insert: [ internal use only ]
|
|
|
|
*
|
1998-12-23 01:52:47 +00:00
|
|
|
* Inserts the given mem entry into the object and object list.
|
|
|
|
*
|
|
|
|
* The pagetables are not updated but will presumably fault the page
|
|
|
|
* in if necessary, or if a kernel page the caller will at some point
|
|
|
|
* enter the page into the kernel's pmap. We are not allowed to block
|
|
|
|
* here so we *can't* do this anyway.
|
1994-05-24 10:09:53 +00:00
|
|
|
*
|
1995-03-01 23:30:04 +00:00
|
|
|
* The object and page must be locked, and must be splhigh.
|
1998-12-23 01:52:47 +00:00
|
|
|
* This routine may not block.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1996-09-08 20:44:49 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_insert(vm_page_t m, vm_object_t object, vm_pindex_t pindex)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2002-10-18 17:24:30 +00:00
|
|
|
vm_page_t root;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2003-04-22 14:26:02 +00:00
|
|
|
if (!VM_OBJECT_LOCKED(object))
|
|
|
|
GIANT_REQUIRED;
|
1998-10-21 14:46:42 +00:00
|
|
|
if (m->object != NULL)
|
1994-05-24 10:09:53 +00:00
|
|
|
panic("vm_page_insert: already inserted");
|
|
|
|
|
|
|
|
/*
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
* Record the object/offset pair in this page
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1996-01-19 04:00:31 +00:00
|
|
|
m->object = object;
|
|
|
|
m->pindex = pindex;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
2002-10-18 17:24:30 +00:00
|
|
|
* Now link into the object's ordered list of backed pages.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2002-11-09 08:27:42 +00:00
|
|
|
root = object->root;
|
2002-10-18 17:24:30 +00:00
|
|
|
if (root == NULL) {
|
|
|
|
m->left = NULL;
|
|
|
|
m->right = NULL;
|
|
|
|
TAILQ_INSERT_TAIL(&object->memq, m, listq);
|
|
|
|
} else {
|
2002-11-09 08:27:42 +00:00
|
|
|
root = vm_page_splay(pindex, root);
|
|
|
|
if (pindex < root->pindex) {
|
|
|
|
m->left = root->left;
|
|
|
|
m->right = root;
|
|
|
|
root->left = NULL;
|
|
|
|
TAILQ_INSERT_BEFORE(root, m, listq);
|
|
|
|
} else {
|
|
|
|
m->right = root->right;
|
|
|
|
m->left = root;
|
|
|
|
root->right = NULL;
|
|
|
|
TAILQ_INSERT_AFTER(&object->memq, root, m, listq);
|
|
|
|
}
|
2002-10-18 17:24:30 +00:00
|
|
|
}
|
|
|
|
object->root = m;
|
1999-07-31 04:19:49 +00:00
|
|
|
object->generation++;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
1999-01-21 08:29:12 +00:00
|
|
|
* show that the object has one more resident page.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
|
|
|
object->resident_page_count++;
|
1999-02-24 21:26:26 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Since we are inserting a new and possibly dirty page,
|
|
|
|
* update the object's OBJ_WRITEABLE and OBJ_MIGHTBEDIRTY flags.
|
|
|
|
*/
|
|
|
|
if (m->flags & PG_WRITEABLE)
|
2001-10-26 00:08:05 +00:00
|
|
|
vm_object_set_writeable_dirty(object);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
1999-01-21 08:29:12 +00:00
|
|
|
* vm_page_remove:
|
1994-05-24 10:09:53 +00:00
|
|
|
* NOTE: used by device pager as well -wfj
|
|
|
|
*
|
|
|
|
* Removes the given mem entry from the object/offset-page
|
1999-01-21 08:29:12 +00:00
|
|
|
* table and the object page list, but do not invalidate/terminate
|
|
|
|
* the backing store.
|
1994-05-24 10:09:53 +00:00
|
|
|
*
|
1995-03-01 23:30:04 +00:00
|
|
|
* The object and page must be locked, and at splhigh.
|
1999-01-21 08:29:12 +00:00
|
|
|
* The underlying pmap entry (if any) is NOT removed here.
|
1998-12-23 01:52:47 +00:00
|
|
|
* This routine may not block.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1999-02-15 06:52:14 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_remove(vm_page_t m)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
1998-02-05 03:32:49 +00:00
|
|
|
vm_object_t object;
|
2002-10-18 17:24:30 +00:00
|
|
|
vm_page_t root;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2002-12-15 00:06:02 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
1998-10-21 14:46:42 +00:00
|
|
|
if (m->object == NULL)
|
1999-02-15 06:52:14 +00:00
|
|
|
return;
|
2003-04-25 06:35:05 +00:00
|
|
|
if (!VM_OBJECT_LOCKED(m->object))
|
|
|
|
GIANT_REQUIRED;
|
1998-01-31 11:56:53 +00:00
|
|
|
if ((m->flags & PG_BUSY) == 0) {
|
|
|
|
panic("vm_page_remove: page not busy");
|
|
|
|
}
|
1999-01-21 08:29:12 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Basically destroy the page.
|
|
|
|
*/
|
|
|
|
vm_page_wakeup(m);
|
1998-01-31 11:56:53 +00:00
|
|
|
|
1998-02-05 03:32:49 +00:00
|
|
|
object = m->object;
|
1996-09-28 03:33:40 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
* Now remove from the object's list of backed pages.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2002-10-18 17:24:30 +00:00
|
|
|
if (m != object->root)
|
|
|
|
vm_page_splay(m->pindex, object->root);
|
|
|
|
if (m->left == NULL)
|
|
|
|
root = m->right;
|
|
|
|
else {
|
|
|
|
root = vm_page_splay(m->pindex, m->left);
|
|
|
|
root->right = m->right;
|
|
|
|
}
|
|
|
|
object->root = root;
|
1998-02-05 03:32:49 +00:00
|
|
|
TAILQ_REMOVE(&object->memq, m, listq);
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
* And show that the object has one fewer resident page.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1998-02-05 03:32:49 +00:00
|
|
|
object->resident_page_count--;
|
|
|
|
object->generation++;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
1998-10-21 14:46:42 +00:00
|
|
|
m->object = NULL;
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_lookup:
|
|
|
|
*
|
|
|
|
* Returns the page associated with the object/offset
|
|
|
|
* pair specified; if none is found, NULL is returned.
|
|
|
|
*
|
2002-10-18 17:24:30 +00:00
|
|
|
* The object must be locked.
|
1998-12-23 01:52:47 +00:00
|
|
|
* This routine may not block.
|
1999-01-21 08:29:12 +00:00
|
|
|
* This is a critical path routine
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1995-05-30 08:16:23 +00:00
|
|
|
vm_page_t
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_lookup(vm_object_t object, vm_pindex_t pindex)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2001-07-04 19:00:13 +00:00
|
|
|
vm_page_t m;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2003-04-22 05:36:14 +00:00
|
|
|
if (!VM_OBJECT_LOCKED(object))
|
|
|
|
GIANT_REQUIRED;
|
2002-10-18 17:24:30 +00:00
|
|
|
m = vm_page_splay(pindex, object->root);
|
|
|
|
if ((object->root = m) != NULL && m->pindex != pindex)
|
|
|
|
m = NULL;
|
2002-04-30 21:24:47 +00:00
|
|
|
return (m);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_rename:
|
|
|
|
*
|
|
|
|
* Move the given memory entry from its
|
|
|
|
* current object to the specified target object/offset.
|
|
|
|
*
|
|
|
|
* The object must be locked.
|
1998-12-23 01:52:47 +00:00
|
|
|
* This routine may not block.
|
|
|
|
*
|
|
|
|
* Note: this routine will raise itself to splvm(), the caller need not.
|
1999-01-21 08:29:12 +00:00
|
|
|
*
|
|
|
|
* Note: swap associated with the page must be invalidated by the move. We
|
|
|
|
* have to do this for several reasons: (1) we aren't freeing the
|
|
|
|
* page, (2) we are dirtying the page, (3) the VM system is probably
|
|
|
|
* moving the page from object A to B, and will then later move
|
|
|
|
* the backing store from A to B and we can't have a conflict.
|
|
|
|
*
|
|
|
|
* Note: we *always* dirty the page. It is necessary both for the
|
|
|
|
* fact that we moved it, and because we may be invalidating
|
1999-01-24 06:00:31 +00:00
|
|
|
* swap. If the page is on the cache, we have to deactivate it
|
|
|
|
* or vm_page_dirty() will panic. Dirty pages are not allowed
|
|
|
|
* on the cache.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1995-05-30 08:16:23 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_rename(vm_page_t m, vm_object_t new_object, vm_pindex_t new_pindex)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
int s;
|
|
|
|
|
1996-05-18 03:38:05 +00:00
|
|
|
s = splvm();
|
1996-01-19 04:00:31 +00:00
|
|
|
vm_page_remove(m);
|
|
|
|
vm_page_insert(m, new_object, new_pindex);
|
1999-01-24 06:00:31 +00:00
|
|
|
if (m->queue - m->pc == PQ_CACHE)
|
|
|
|
vm_page_deactivate(m);
|
|
|
|
vm_page_dirty(m);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
splx(s);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
This mega-commit is meant to fix numerous interrelated problems. There
has been some bitrot and incorrect assumptions in the vfs_bio code. These
problems have manifest themselves worse on NFS type filesystems, but can
still affect local filesystems under certain circumstances. Most of
the problems have involved mmap consistancy, and as a side-effect broke
the vfs.ioopt code. This code might have been committed seperately, but
almost everything is interrelated.
1) Allow (pmap_object_init_pt) prefaulting of buffer-busy pages that
are fully valid.
2) Rather than deactivating erroneously read initial (header) pages in
kern_exec, we now free them.
3) Fix the rundown of non-VMIO buffers that are in an inconsistent
(missing vp) state.
4) Fix the disassociation of pages from buffers in brelse. The previous
code had rotted and was faulty in a couple of important circumstances.
5) Remove a gratuitious buffer wakeup in vfs_vmio_release.
6) Remove a crufty and currently unused cluster mechanism for VBLK
files in vfs_bio_awrite. When the code is functional, I'll add back
a cleaner version.
7) The page busy count wakeups assocated with the buffer cache usage were
incorrectly cleaned up in a previous commit by me. Revert to the
original, correct version, but with a cleaner implementation.
8) The cluster read code now tries to keep data associated with buffers
more aggressively (without breaking the heuristics) when it is presumed
that the read data (buffers) will be soon needed.
9) Change to filesystem lockmgr locks so that they use LK_NOPAUSE. The
delay loop waiting is not useful for filesystem locks, due to the
length of the time intervals.
10) Correct and clean-up spec_getpages.
11) Implement a fully functional nfs_getpages, nfs_putpages.
12) Fix nfs_write so that modifications are coherent with the NFS data on
the server disk (at least as well as NFS seems to allow.)
13) Properly support MS_INVALIDATE on NFS.
14) Properly pass down MS_INVALIDATE to lower levels of the VM code from
vm_map_clean.
15) Better support the notion of pages being busy but valid, so that
fewer in-transit waits occur. (use p->busy more for pageouts instead
of PG_BUSY.) Since the page is fully valid, it is still usable for
reads.
16) It is possible (in error) for cached pages to be busy. Make the
page allocation code handle that case correctly. (It should probably
be a printf or panic, but I want the system to handle coding errors
robustly. I'll probably add a printf.)
17) Correct the design and usage of vm_page_sleep. It didn't handle
consistancy problems very well, so make the design a little less
lofty. After vm_page_sleep, if it ever blocked, it is still important
to relookup the page (if the object generation count changed), and
verify it's status (always.)
18) In vm_pageout.c, vm_pageout_clean had rotted, so clean that up.
19) Push the page busy for writes and VM_PROT_READ into vm_pageout_flush.
20) Fix vm_pager_put_pages and it's descendents to support an int flag
instead of a boolean, so that we can pass down the invalidate bit.
1998-03-07 21:37:31 +00:00
|
|
|
/*
|
1998-12-23 01:52:47 +00:00
|
|
|
* vm_page_select_cache:
|
|
|
|
*
|
|
|
|
* Find a page on the cache queue with color optimization. As pages
|
|
|
|
* might be found, but not applicable, they are deactivated. This
|
|
|
|
* keeps us from using potentially busy cached pages.
|
|
|
|
*
|
|
|
|
* This routine must be called at splvm().
|
|
|
|
* This routine may not block.
|
This mega-commit is meant to fix numerous interrelated problems. There
has been some bitrot and incorrect assumptions in the vfs_bio code. These
problems have manifest themselves worse on NFS type filesystems, but can
still affect local filesystems under certain circumstances. Most of
the problems have involved mmap consistancy, and as a side-effect broke
the vfs.ioopt code. This code might have been committed seperately, but
almost everything is interrelated.
1) Allow (pmap_object_init_pt) prefaulting of buffer-busy pages that
are fully valid.
2) Rather than deactivating erroneously read initial (header) pages in
kern_exec, we now free them.
3) Fix the rundown of non-VMIO buffers that are in an inconsistent
(missing vp) state.
4) Fix the disassociation of pages from buffers in brelse. The previous
code had rotted and was faulty in a couple of important circumstances.
5) Remove a gratuitious buffer wakeup in vfs_vmio_release.
6) Remove a crufty and currently unused cluster mechanism for VBLK
files in vfs_bio_awrite. When the code is functional, I'll add back
a cleaner version.
7) The page busy count wakeups assocated with the buffer cache usage were
incorrectly cleaned up in a previous commit by me. Revert to the
original, correct version, but with a cleaner implementation.
8) The cluster read code now tries to keep data associated with buffers
more aggressively (without breaking the heuristics) when it is presumed
that the read data (buffers) will be soon needed.
9) Change to filesystem lockmgr locks so that they use LK_NOPAUSE. The
delay loop waiting is not useful for filesystem locks, due to the
length of the time intervals.
10) Correct and clean-up spec_getpages.
11) Implement a fully functional nfs_getpages, nfs_putpages.
12) Fix nfs_write so that modifications are coherent with the NFS data on
the server disk (at least as well as NFS seems to allow.)
13) Properly support MS_INVALIDATE on NFS.
14) Properly pass down MS_INVALIDATE to lower levels of the VM code from
vm_map_clean.
15) Better support the notion of pages being busy but valid, so that
fewer in-transit waits occur. (use p->busy more for pageouts instead
of PG_BUSY.) Since the page is fully valid, it is still usable for
reads.
16) It is possible (in error) for cached pages to be busy. Make the
page allocation code handle that case correctly. (It should probably
be a printf or panic, but I want the system to handle coding errors
robustly. I'll probably add a printf.)
17) Correct the design and usage of vm_page_sleep. It didn't handle
consistancy problems very well, so make the design a little less
lofty. After vm_page_sleep, if it ever blocked, it is still important
to relookup the page (if the object generation count changed), and
verify it's status (always.)
18) In vm_pageout.c, vm_pageout_clean had rotted, so clean that up.
19) Push the page busy for writes and VM_PROT_READ into vm_pageout_flush.
20) Fix vm_pager_put_pages and it's descendents to support an int flag
instead of a boolean, so that we can pass down the invalidate bit.
1998-03-07 21:37:31 +00:00
|
|
|
*/
|
2001-07-23 12:34:31 +00:00
|
|
|
static vm_page_t
|
2002-11-01 00:59:03 +00:00
|
|
|
vm_page_select_cache(vm_pindex_t color)
|
This mega-commit is meant to fix numerous interrelated problems. There
has been some bitrot and incorrect assumptions in the vfs_bio code. These
problems have manifest themselves worse on NFS type filesystems, but can
still affect local filesystems under certain circumstances. Most of
the problems have involved mmap consistancy, and as a side-effect broke
the vfs.ioopt code. This code might have been committed seperately, but
almost everything is interrelated.
1) Allow (pmap_object_init_pt) prefaulting of buffer-busy pages that
are fully valid.
2) Rather than deactivating erroneously read initial (header) pages in
kern_exec, we now free them.
3) Fix the rundown of non-VMIO buffers that are in an inconsistent
(missing vp) state.
4) Fix the disassociation of pages from buffers in brelse. The previous
code had rotted and was faulty in a couple of important circumstances.
5) Remove a gratuitious buffer wakeup in vfs_vmio_release.
6) Remove a crufty and currently unused cluster mechanism for VBLK
files in vfs_bio_awrite. When the code is functional, I'll add back
a cleaner version.
7) The page busy count wakeups assocated with the buffer cache usage were
incorrectly cleaned up in a previous commit by me. Revert to the
original, correct version, but with a cleaner implementation.
8) The cluster read code now tries to keep data associated with buffers
more aggressively (without breaking the heuristics) when it is presumed
that the read data (buffers) will be soon needed.
9) Change to filesystem lockmgr locks so that they use LK_NOPAUSE. The
delay loop waiting is not useful for filesystem locks, due to the
length of the time intervals.
10) Correct and clean-up spec_getpages.
11) Implement a fully functional nfs_getpages, nfs_putpages.
12) Fix nfs_write so that modifications are coherent with the NFS data on
the server disk (at least as well as NFS seems to allow.)
13) Properly support MS_INVALIDATE on NFS.
14) Properly pass down MS_INVALIDATE to lower levels of the VM code from
vm_map_clean.
15) Better support the notion of pages being busy but valid, so that
fewer in-transit waits occur. (use p->busy more for pageouts instead
of PG_BUSY.) Since the page is fully valid, it is still usable for
reads.
16) It is possible (in error) for cached pages to be busy. Make the
page allocation code handle that case correctly. (It should probably
be a printf or panic, but I want the system to handle coding errors
robustly. I'll probably add a printf.)
17) Correct the design and usage of vm_page_sleep. It didn't handle
consistancy problems very well, so make the design a little less
lofty. After vm_page_sleep, if it ever blocked, it is still important
to relookup the page (if the object generation count changed), and
verify it's status (always.)
18) In vm_pageout.c, vm_pageout_clean had rotted, so clean that up.
19) Push the page busy for writes and VM_PROT_READ into vm_pageout_flush.
20) Fix vm_pager_put_pages and it's descendents to support an int flag
instead of a boolean, so that we can pass down the invalidate bit.
1998-03-07 21:37:31 +00:00
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
2002-07-04 22:07:37 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
This mega-commit is meant to fix numerous interrelated problems. There
has been some bitrot and incorrect assumptions in the vfs_bio code. These
problems have manifest themselves worse on NFS type filesystems, but can
still affect local filesystems under certain circumstances. Most of
the problems have involved mmap consistancy, and as a side-effect broke
the vfs.ioopt code. This code might have been committed seperately, but
almost everything is interrelated.
1) Allow (pmap_object_init_pt) prefaulting of buffer-busy pages that
are fully valid.
2) Rather than deactivating erroneously read initial (header) pages in
kern_exec, we now free them.
3) Fix the rundown of non-VMIO buffers that are in an inconsistent
(missing vp) state.
4) Fix the disassociation of pages from buffers in brelse. The previous
code had rotted and was faulty in a couple of important circumstances.
5) Remove a gratuitious buffer wakeup in vfs_vmio_release.
6) Remove a crufty and currently unused cluster mechanism for VBLK
files in vfs_bio_awrite. When the code is functional, I'll add back
a cleaner version.
7) The page busy count wakeups assocated with the buffer cache usage were
incorrectly cleaned up in a previous commit by me. Revert to the
original, correct version, but with a cleaner implementation.
8) The cluster read code now tries to keep data associated with buffers
more aggressively (without breaking the heuristics) when it is presumed
that the read data (buffers) will be soon needed.
9) Change to filesystem lockmgr locks so that they use LK_NOPAUSE. The
delay loop waiting is not useful for filesystem locks, due to the
length of the time intervals.
10) Correct and clean-up spec_getpages.
11) Implement a fully functional nfs_getpages, nfs_putpages.
12) Fix nfs_write so that modifications are coherent with the NFS data on
the server disk (at least as well as NFS seems to allow.)
13) Properly support MS_INVALIDATE on NFS.
14) Properly pass down MS_INVALIDATE to lower levels of the VM code from
vm_map_clean.
15) Better support the notion of pages being busy but valid, so that
fewer in-transit waits occur. (use p->busy more for pageouts instead
of PG_BUSY.) Since the page is fully valid, it is still usable for
reads.
16) It is possible (in error) for cached pages to be busy. Make the
page allocation code handle that case correctly. (It should probably
be a printf or panic, but I want the system to handle coding errors
robustly. I'll probably add a printf.)
17) Correct the design and usage of vm_page_sleep. It didn't handle
consistancy problems very well, so make the design a little less
lofty. After vm_page_sleep, if it ever blocked, it is still important
to relookup the page (if the object generation count changed), and
verify it's status (always.)
18) In vm_pageout.c, vm_pageout_clean had rotted, so clean that up.
19) Push the page busy for writes and VM_PROT_READ into vm_pageout_flush.
20) Fix vm_pager_put_pages and it's descendents to support an int flag
instead of a boolean, so that we can pass down the invalidate bit.
1998-03-07 21:37:31 +00:00
|
|
|
while (TRUE) {
|
2002-11-01 00:59:03 +00:00
|
|
|
m = vm_pageq_find(PQ_CACHE, color & PQ_L2_MASK, FALSE);
|
2000-05-29 22:40:54 +00:00
|
|
|
if (m && ((m->flags & (PG_BUSY|PG_UNMANAGED)) || m->busy ||
|
This mega-commit is meant to fix numerous interrelated problems. There
has been some bitrot and incorrect assumptions in the vfs_bio code. These
problems have manifest themselves worse on NFS type filesystems, but can
still affect local filesystems under certain circumstances. Most of
the problems have involved mmap consistancy, and as a side-effect broke
the vfs.ioopt code. This code might have been committed seperately, but
almost everything is interrelated.
1) Allow (pmap_object_init_pt) prefaulting of buffer-busy pages that
are fully valid.
2) Rather than deactivating erroneously read initial (header) pages in
kern_exec, we now free them.
3) Fix the rundown of non-VMIO buffers that are in an inconsistent
(missing vp) state.
4) Fix the disassociation of pages from buffers in brelse. The previous
code had rotted and was faulty in a couple of important circumstances.
5) Remove a gratuitious buffer wakeup in vfs_vmio_release.
6) Remove a crufty and currently unused cluster mechanism for VBLK
files in vfs_bio_awrite. When the code is functional, I'll add back
a cleaner version.
7) The page busy count wakeups assocated with the buffer cache usage were
incorrectly cleaned up in a previous commit by me. Revert to the
original, correct version, but with a cleaner implementation.
8) The cluster read code now tries to keep data associated with buffers
more aggressively (without breaking the heuristics) when it is presumed
that the read data (buffers) will be soon needed.
9) Change to filesystem lockmgr locks so that they use LK_NOPAUSE. The
delay loop waiting is not useful for filesystem locks, due to the
length of the time intervals.
10) Correct and clean-up spec_getpages.
11) Implement a fully functional nfs_getpages, nfs_putpages.
12) Fix nfs_write so that modifications are coherent with the NFS data on
the server disk (at least as well as NFS seems to allow.)
13) Properly support MS_INVALIDATE on NFS.
14) Properly pass down MS_INVALIDATE to lower levels of the VM code from
vm_map_clean.
15) Better support the notion of pages being busy but valid, so that
fewer in-transit waits occur. (use p->busy more for pageouts instead
of PG_BUSY.) Since the page is fully valid, it is still usable for
reads.
16) It is possible (in error) for cached pages to be busy. Make the
page allocation code handle that case correctly. (It should probably
be a printf or panic, but I want the system to handle coding errors
robustly. I'll probably add a printf.)
17) Correct the design and usage of vm_page_sleep. It didn't handle
consistancy problems very well, so make the design a little less
lofty. After vm_page_sleep, if it ever blocked, it is still important
to relookup the page (if the object generation count changed), and
verify it's status (always.)
18) In vm_pageout.c, vm_pageout_clean had rotted, so clean that up.
19) Push the page busy for writes and VM_PROT_READ into vm_pageout_flush.
20) Fix vm_pager_put_pages and it's descendents to support an int flag
instead of a boolean, so that we can pass down the invalidate bit.
1998-03-07 21:37:31 +00:00
|
|
|
m->hold_count || m->wire_count)) {
|
|
|
|
vm_page_deactivate(m);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
return m;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
1996-09-08 20:44:49 +00:00
|
|
|
/*
|
1998-12-23 01:52:47 +00:00
|
|
|
* vm_page_select_free:
|
|
|
|
*
|
2001-07-04 20:15:18 +00:00
|
|
|
* Find a free or zero page, with specified preference.
|
1998-12-23 01:52:47 +00:00
|
|
|
*
|
|
|
|
* This routine must be called at splvm().
|
|
|
|
* This routine may not block.
|
1996-09-08 20:44:49 +00:00
|
|
|
*/
|
1999-02-07 20:45:15 +00:00
|
|
|
static __inline vm_page_t
|
2002-11-01 00:59:03 +00:00
|
|
|
vm_page_select_free(vm_pindex_t color, boolean_t prefer_zero)
|
1996-09-08 20:44:49 +00:00
|
|
|
{
|
1999-02-08 00:37:36 +00:00
|
|
|
vm_page_t m;
|
1997-11-06 08:35:50 +00:00
|
|
|
|
2002-11-01 00:59:03 +00:00
|
|
|
m = vm_pageq_find(PQ_FREE, color & PQ_L2_MASK, prefer_zero);
|
2002-03-10 21:52:48 +00:00
|
|
|
return (m);
|
1999-02-07 20:45:15 +00:00
|
|
|
}
|
1997-11-06 08:35:50 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* vm_page_alloc:
|
|
|
|
*
|
|
|
|
* Allocate and return a memory cell associated
|
|
|
|
* with this VM object/offset pair.
|
|
|
|
*
|
1995-03-01 23:30:04 +00:00
|
|
|
* page_req classes:
|
|
|
|
* VM_ALLOC_NORMAL normal process request
|
|
|
|
* VM_ALLOC_SYSTEM system *really* needs a page
|
|
|
|
* VM_ALLOC_INTERRUPT interrupt time request
|
1995-09-03 20:40:43 +00:00
|
|
|
* VM_ALLOC_ZERO zero page
|
1995-01-24 10:14:09 +00:00
|
|
|
*
|
1998-12-23 01:52:47 +00:00
|
|
|
* This routine may not block.
|
|
|
|
*
|
|
|
|
* Additional special handling is required when called from an
|
|
|
|
* interrupt (VM_ALLOC_INTERRUPT). We are not allowed to mess with
|
|
|
|
* the page cache in this case.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1994-05-25 09:21:21 +00:00
|
|
|
vm_page_t
|
2002-07-18 04:08:10 +00:00
|
|
|
vm_page_alloc(vm_object_t object, vm_pindex_t pindex, int req)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2001-07-04 19:00:13 +00:00
|
|
|
vm_page_t m = NULL;
|
2002-11-01 00:59:03 +00:00
|
|
|
vm_pindex_t color;
|
2003-01-12 23:32:46 +00:00
|
|
|
int flags, page_req, s;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2003-01-11 20:07:17 +00:00
|
|
|
page_req = req & VM_ALLOC_CLASS_MASK;
|
|
|
|
|
2002-11-01 00:59:03 +00:00
|
|
|
if ((req & VM_ALLOC_NOOBJ) == 0) {
|
|
|
|
KASSERT(object != NULL,
|
|
|
|
("vm_page_alloc: NULL object."));
|
|
|
|
KASSERT(!vm_page_lookup(object, pindex),
|
|
|
|
("vm_page_alloc: page already allocated"));
|
|
|
|
color = pindex + object->pg_color;
|
2003-01-11 20:07:17 +00:00
|
|
|
} else
|
2002-11-01 00:59:03 +00:00
|
|
|
color = pindex;
|
|
|
|
|
1999-01-21 08:29:12 +00:00
|
|
|
/*
|
|
|
|
* The pager is allowed to eat deeper into the free page list.
|
|
|
|
*/
|
1995-03-01 23:30:04 +00:00
|
|
|
if ((curproc == pageproc) && (page_req != VM_ALLOC_INTERRUPT)) {
|
|
|
|
page_req = VM_ALLOC_SYSTEM;
|
|
|
|
};
|
1995-05-30 08:16:23 +00:00
|
|
|
|
1996-05-18 03:38:05 +00:00
|
|
|
s = splvm();
|
1999-01-21 08:29:12 +00:00
|
|
|
loop:
|
2002-07-04 22:07:37 +00:00
|
|
|
mtx_lock_spin(&vm_page_queue_free_mtx);
|
2003-01-08 19:58:42 +00:00
|
|
|
if (cnt.v_free_count > cnt.v_free_reserved ||
|
1999-02-15 06:52:14 +00:00
|
|
|
(page_req == VM_ALLOC_SYSTEM &&
|
|
|
|
cnt.v_cache_count == 0 &&
|
|
|
|
cnt.v_free_count > cnt.v_interrupt_free_min) ||
|
2003-01-08 19:58:42 +00:00
|
|
|
(page_req == VM_ALLOC_INTERRUPT && cnt.v_free_count > 0)) {
|
1999-02-15 06:52:14 +00:00
|
|
|
/*
|
2003-01-08 19:58:42 +00:00
|
|
|
* Allocate from the free queue if the number of free pages
|
|
|
|
* exceeds the minimum for the request class.
|
1999-02-15 06:52:14 +00:00
|
|
|
*/
|
2003-01-08 19:58:42 +00:00
|
|
|
m = vm_page_select_free(color, (req & VM_ALLOC_ZERO) != 0);
|
1999-02-15 06:52:14 +00:00
|
|
|
} else if (page_req != VM_ALLOC_INTERRUPT) {
|
2002-07-04 22:07:37 +00:00
|
|
|
mtx_unlock_spin(&vm_page_queue_free_mtx);
|
1999-02-15 06:52:14 +00:00
|
|
|
/*
|
2000-03-26 15:20:23 +00:00
|
|
|
* Allocatable from cache (non-interrupt only). On success,
|
1999-02-15 06:52:14 +00:00
|
|
|
* we must free the page and try again, thus ensuring that
|
|
|
|
* cnt.v_*_free_min counters are replenished.
|
|
|
|
*/
|
2002-07-04 22:07:37 +00:00
|
|
|
vm_page_lock_queues();
|
2002-11-01 00:59:03 +00:00
|
|
|
if ((m = vm_page_select_cache(color)) == NULL) {
|
2002-07-04 22:07:37 +00:00
|
|
|
vm_page_unlock_queues();
|
1999-02-15 06:52:14 +00:00
|
|
|
splx(s);
|
1996-05-31 00:38:04 +00:00
|
|
|
#if defined(DIAGNOSTIC)
|
1999-02-15 06:52:14 +00:00
|
|
|
if (cnt.v_cache_count > 0)
|
|
|
|
printf("vm_page_alloc(NORMAL): missing pages on cache queue: %d\n", cnt.v_cache_count);
|
1996-05-31 00:38:04 +00:00
|
|
|
#endif
|
2003-01-14 06:57:03 +00:00
|
|
|
atomic_add_int(&vm_pageout_deficit, 1);
|
1995-03-01 23:30:04 +00:00
|
|
|
pagedaemon_wakeup();
|
1996-01-19 04:00:31 +00:00
|
|
|
return (NULL);
|
1995-03-01 23:30:04 +00:00
|
|
|
}
|
1999-02-15 06:52:14 +00:00
|
|
|
KASSERT(m->dirty == 0, ("Found dirty cache page %p", m));
|
|
|
|
vm_page_busy(m);
|
2002-11-16 07:44:25 +00:00
|
|
|
pmap_remove_all(m);
|
1999-02-15 06:52:14 +00:00
|
|
|
vm_page_free(m);
|
2002-07-04 22:07:37 +00:00
|
|
|
vm_page_unlock_queues();
|
1999-02-15 06:52:14 +00:00
|
|
|
goto loop;
|
|
|
|
} else {
|
|
|
|
/*
|
2000-03-26 15:20:23 +00:00
|
|
|
* Not allocatable from cache from interrupt, give up.
|
1999-02-15 06:52:14 +00:00
|
|
|
*/
|
2002-07-04 22:07:37 +00:00
|
|
|
mtx_unlock_spin(&vm_page_queue_free_mtx);
|
1999-02-15 06:52:14 +00:00
|
|
|
splx(s);
|
2003-01-14 06:57:03 +00:00
|
|
|
atomic_add_int(&vm_pageout_deficit, 1);
|
1999-02-15 06:52:14 +00:00
|
|
|
pagedaemon_wakeup();
|
|
|
|
return (NULL);
|
1995-03-01 23:30:04 +00:00
|
|
|
}
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
1999-02-15 06:52:14 +00:00
|
|
|
/*
|
|
|
|
* At this point we had better have found a good page.
|
|
|
|
*/
|
|
|
|
|
|
|
|
KASSERT(
|
|
|
|
m != NULL,
|
|
|
|
("vm_page_alloc(): missing page on free queue\n")
|
|
|
|
);
|
1999-01-21 08:29:12 +00:00
|
|
|
|
|
|
|
/*
|
1999-02-15 06:52:14 +00:00
|
|
|
* Remove from free queue
|
1999-01-21 08:29:12 +00:00
|
|
|
*/
|
|
|
|
|
2001-07-04 23:27:09 +00:00
|
|
|
vm_pageq_remove_nowakeup(m);
|
1999-01-21 08:29:12 +00:00
|
|
|
|
1999-02-15 06:52:14 +00:00
|
|
|
/*
|
|
|
|
* Initialize structure. Only the PG_ZERO flag is inherited.
|
|
|
|
*/
|
2003-01-12 23:32:46 +00:00
|
|
|
flags = PG_BUSY;
|
1999-02-08 00:37:36 +00:00
|
|
|
if (m->flags & PG_ZERO) {
|
1999-01-21 08:29:12 +00:00
|
|
|
vm_page_zero_count--;
|
2003-01-12 23:32:46 +00:00
|
|
|
if (req & VM_ALLOC_ZERO)
|
|
|
|
flags = PG_ZERO | PG_BUSY;
|
1996-01-19 04:00:31 +00:00
|
|
|
}
|
2003-01-12 23:32:46 +00:00
|
|
|
m->flags = flags;
|
2002-07-18 04:08:10 +00:00
|
|
|
if (req & VM_ALLOC_WIRED) {
|
2003-01-05 01:31:45 +00:00
|
|
|
atomic_add_int(&cnt.v_wire_count, 1);
|
2002-07-18 04:08:10 +00:00
|
|
|
m->wire_count = 1;
|
|
|
|
} else
|
|
|
|
m->wire_count = 0;
|
1996-05-18 03:38:05 +00:00
|
|
|
m->hold_count = 0;
|
This commit does a couple of things:
Re-enables the RSS limiting, and the routine is now tail-recursive,
making it much more safe (eliminates the possiblity of kernel stack
overflow.) Also, the RSS limiting is a little more intelligent about
finding the likely objects that are pushing the process over the limit.
Added some sysctls that help with VM system tuning.
New sysctl features:
1) Enable/disable lru pageout algorithm.
vm.pageout_algorithm = 0, default algorithm that works
well, especially using X windows and heavy
memory loading. Can have adverse effects,
sometimes slowing down program loading.
vm.pageout_algorithm = 1, close to true LRU. Works much
better than clock, etc. Does not work as well as
the default algorithm in general. Certain memory
"malloc" type benchmarks work a little better with
this setting.
Please give me feedback on the performance results
associated with these.
2) Enable/disable swapping.
vm.swapping_enabled = 1, default.
vm.swapping_enabled = 0, useful for cases where swapping
degrades performance.
The config option "NO_SWAPPING" is still operative, and
takes precedence over the sysctl. If "NO_SWAPPING" is
specified, the sysctl still exists, but "vm.swapping_enabled"
is hard-wired to "0".
Each of these can be changed "on the fly."
1996-06-26 05:39:27 +00:00
|
|
|
m->act_count = 0;
|
1996-01-19 04:00:31 +00:00
|
|
|
m->busy = 0;
|
|
|
|
m->valid = 0;
|
1999-08-20 06:32:00 +00:00
|
|
|
KASSERT(m->dirty == 0, ("vm_page_alloc: free/cache page %p was dirty", m));
|
2002-07-04 22:07:37 +00:00
|
|
|
mtx_unlock_spin(&vm_page_queue_free_mtx);
|
1995-01-10 09:19:52 +00:00
|
|
|
|
1998-12-23 01:52:47 +00:00
|
|
|
/*
|
|
|
|
* vm_page_insert() is safe prior to the splx(). Note also that
|
|
|
|
* inserting a page here does not insert it into the pmap (which
|
|
|
|
* could cause us to block allocating memory). We cannot block
|
|
|
|
* anywhere.
|
|
|
|
*/
|
2002-11-01 00:59:03 +00:00
|
|
|
if ((req & VM_ALLOC_NOOBJ) == 0)
|
|
|
|
vm_page_insert(m, object, pindex);
|
1995-01-10 09:19:52 +00:00
|
|
|
|
1995-03-01 23:30:04 +00:00
|
|
|
/*
|
|
|
|
* Don't wakeup too often - wakeup the pageout daemon when
|
|
|
|
* we would be nearly out of memory.
|
|
|
|
*/
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
if (vm_paging_needed())
|
1995-03-01 23:30:04 +00:00
|
|
|
pagedaemon_wakeup();
|
1994-05-24 10:09:53 +00:00
|
|
|
|
1998-01-17 09:17:02 +00:00
|
|
|
splx(s);
|
1996-01-19 04:00:31 +00:00
|
|
|
return (m);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
1998-12-23 01:52:47 +00:00
|
|
|
/*
|
|
|
|
* vm_wait: (also see VM_WAIT macro)
|
|
|
|
*
|
|
|
|
* Block until free pages are available for allocation
|
2002-02-19 18:34:02 +00:00
|
|
|
* - Called in various places before memory allocations.
|
1998-12-23 01:52:47 +00:00
|
|
|
*/
|
1996-11-28 23:15:07 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_wait(void)
|
1996-11-28 23:15:07 +00:00
|
|
|
{
|
|
|
|
int s;
|
|
|
|
|
|
|
|
s = splvm();
|
2003-02-01 21:18:16 +00:00
|
|
|
vm_page_lock_queues();
|
1996-11-28 23:15:07 +00:00
|
|
|
if (curproc == pageproc) {
|
|
|
|
vm_pageout_pages_needed = 1;
|
2003-02-01 21:18:16 +00:00
|
|
|
msleep(&vm_pageout_pages_needed, &vm_page_queue_mtx,
|
|
|
|
PDROP | PSWP, "VMWait", 0);
|
1996-11-28 23:15:07 +00:00
|
|
|
} else {
|
|
|
|
if (!vm_pages_needed) {
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
vm_pages_needed = 1;
|
1996-11-28 23:15:07 +00:00
|
|
|
wakeup(&vm_pages_needed);
|
|
|
|
}
|
2003-02-01 21:18:16 +00:00
|
|
|
msleep(&cnt.v_free_count, &vm_page_queue_mtx, PDROP | PVM,
|
|
|
|
"vmwait", 0);
|
1996-11-28 23:15:07 +00:00
|
|
|
}
|
|
|
|
splx(s);
|
|
|
|
}
|
|
|
|
|
2002-02-19 18:34:02 +00:00
|
|
|
/*
|
|
|
|
* vm_waitpfault: (also see VM_WAITPFAULT macro)
|
|
|
|
*
|
|
|
|
* Block until free pages are available for allocation
|
|
|
|
* - Called only in vm_fault so that processes page faulting
|
|
|
|
* can be easily tracked.
|
2002-02-19 18:50:49 +00:00
|
|
|
* - Sleeps at a lower priority than vm_wait() so that vm_wait()ing
|
|
|
|
* processes will be able to grab memory first. Do not change
|
|
|
|
* this balance without careful testing first.
|
2002-02-19 18:34:02 +00:00
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_waitpfault(void)
|
|
|
|
{
|
|
|
|
int s;
|
|
|
|
|
|
|
|
s = splvm();
|
2003-02-01 21:18:16 +00:00
|
|
|
vm_page_lock_queues();
|
2002-02-19 18:34:02 +00:00
|
|
|
if (!vm_pages_needed) {
|
|
|
|
vm_pages_needed = 1;
|
|
|
|
wakeup(&vm_pages_needed);
|
|
|
|
}
|
2003-02-01 21:18:16 +00:00
|
|
|
msleep(&cnt.v_free_count, &vm_page_queue_mtx, PDROP | PUSER,
|
|
|
|
"pfault", 0);
|
2002-02-19 18:34:02 +00:00
|
|
|
splx(s);
|
|
|
|
}
|
|
|
|
|
1996-01-27 00:13:33 +00:00
|
|
|
/*
|
1996-05-18 03:38:05 +00:00
|
|
|
* vm_page_activate:
|
|
|
|
*
|
|
|
|
* Put the specified page on the active list (if appropriate).
|
1999-09-17 04:56:40 +00:00
|
|
|
* Ensure that act_count is at least ACT_INIT but do not otherwise
|
|
|
|
* mess with it.
|
1996-05-18 03:38:05 +00:00
|
|
|
*
|
|
|
|
* The page queues must be locked.
|
1998-12-23 01:52:47 +00:00
|
|
|
* This routine may not block.
|
1996-01-27 00:13:33 +00:00
|
|
|
*/
|
1996-05-18 03:38:05 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_activate(vm_page_t m)
|
1994-09-27 18:00:29 +00:00
|
|
|
{
|
1996-05-18 03:38:05 +00:00
|
|
|
int s;
|
1994-09-27 18:00:29 +00:00
|
|
|
|
2002-08-11 00:21:40 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
1996-05-18 03:38:05 +00:00
|
|
|
s = splvm();
|
1998-01-31 11:56:53 +00:00
|
|
|
if (m->queue != PQ_ACTIVE) {
|
|
|
|
if ((m->queue - m->pc) == PQ_CACHE)
|
|
|
|
cnt.v_reactivated++;
|
2001-07-04 23:27:09 +00:00
|
|
|
vm_pageq_remove(m);
|
2000-05-29 22:40:54 +00:00
|
|
|
if (m->wire_count == 0 && (m->flags & PG_UNMANAGED) == 0) {
|
1998-01-31 11:56:53 +00:00
|
|
|
if (m->act_count < ACT_INIT)
|
|
|
|
m->act_count = ACT_INIT;
|
2002-03-04 18:55:26 +00:00
|
|
|
vm_pageq_enqueue(PQ_ACTIVE, m);
|
1998-01-31 11:56:53 +00:00
|
|
|
}
|
|
|
|
} else {
|
This commit does a couple of things:
Re-enables the RSS limiting, and the routine is now tail-recursive,
making it much more safe (eliminates the possiblity of kernel stack
overflow.) Also, the RSS limiting is a little more intelligent about
finding the likely objects that are pushing the process over the limit.
Added some sysctls that help with VM system tuning.
New sysctl features:
1) Enable/disable lru pageout algorithm.
vm.pageout_algorithm = 0, default algorithm that works
well, especially using X windows and heavy
memory loading. Can have adverse effects,
sometimes slowing down program loading.
vm.pageout_algorithm = 1, close to true LRU. Works much
better than clock, etc. Does not work as well as
the default algorithm in general. Certain memory
"malloc" type benchmarks work a little better with
this setting.
Please give me feedback on the performance results
associated with these.
2) Enable/disable swapping.
vm.swapping_enabled = 1, default.
vm.swapping_enabled = 0, useful for cases where swapping
degrades performance.
The config option "NO_SWAPPING" is still operative, and
takes precedence over the sysctl. If "NO_SWAPPING" is
specified, the sysctl still exists, but "vm.swapping_enabled"
is hard-wired to "0".
Each of these can be changed "on the fly."
1996-06-26 05:39:27 +00:00
|
|
|
if (m->act_count < ACT_INIT)
|
|
|
|
m->act_count = ACT_INIT;
|
1994-09-27 18:00:29 +00:00
|
|
|
}
|
|
|
|
splx(s);
|
|
|
|
}
|
|
|
|
|
1996-06-05 03:31:49 +00:00
|
|
|
/*
|
1999-02-15 06:52:14 +00:00
|
|
|
* vm_page_free_wakeup:
|
|
|
|
*
|
|
|
|
* Helper routine for vm_page_free_toq() and vm_page_cache(). This
|
|
|
|
* routine is called when a page has been added to the cache or free
|
|
|
|
* queues.
|
1998-12-23 01:52:47 +00:00
|
|
|
*
|
1999-02-15 06:52:14 +00:00
|
|
|
* This routine may not block.
|
|
|
|
* This routine must be called at splvm()
|
1996-06-05 03:31:49 +00:00
|
|
|
*/
|
1999-01-21 08:29:12 +00:00
|
|
|
static __inline void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_free_wakeup(void)
|
1996-06-05 03:31:49 +00:00
|
|
|
{
|
2003-02-01 21:18:16 +00:00
|
|
|
|
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
1999-01-21 08:29:12 +00:00
|
|
|
/*
|
|
|
|
* if pageout daemon needs pages, then tell it that there are
|
|
|
|
* some free.
|
|
|
|
*/
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
if (vm_pageout_pages_needed &&
|
|
|
|
cnt.v_cache_count + cnt.v_free_count >= cnt.v_pageout_free_min) {
|
1999-01-21 08:29:12 +00:00
|
|
|
wakeup(&vm_pageout_pages_needed);
|
|
|
|
vm_pageout_pages_needed = 0;
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* wakeup processes that are waiting on memory if we hit a
|
|
|
|
* high water mark. And wakeup scheduler process if we have
|
|
|
|
* lots of memory. this process will swapin processes.
|
|
|
|
*/
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
if (vm_pages_needed && !vm_page_count_min()) {
|
1999-01-21 08:29:12 +00:00
|
|
|
vm_pages_needed = 0;
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
wakeup(&cnt.v_free_count);
|
1999-01-21 08:29:12 +00:00
|
|
|
}
|
|
|
|
}
|
1998-01-17 09:17:02 +00:00
|
|
|
|
1999-01-21 08:29:12 +00:00
|
|
|
/*
|
|
|
|
* vm_page_free_toq:
|
|
|
|
*
|
1999-08-17 18:09:01 +00:00
|
|
|
* Returns the given page to the PQ_FREE list,
|
1999-01-21 08:29:12 +00:00
|
|
|
* disassociating it with any VM object.
|
|
|
|
*
|
|
|
|
* Object and page must be locked prior to entry.
|
|
|
|
* This routine may not block.
|
|
|
|
*/
|
|
|
|
|
|
|
|
void
|
1999-02-08 00:37:36 +00:00
|
|
|
vm_page_free_toq(vm_page_t m)
|
1999-01-21 08:29:12 +00:00
|
|
|
{
|
|
|
|
int s;
|
|
|
|
struct vpgqueues *pq;
|
|
|
|
vm_object_t object = m->object;
|
|
|
|
|
2002-12-15 00:06:02 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
1999-01-21 08:29:12 +00:00
|
|
|
s = splvm();
|
|
|
|
cnt.v_tfree++;
|
1998-01-17 09:17:02 +00:00
|
|
|
|
2002-02-19 23:19:30 +00:00
|
|
|
if (m->busy || ((m->queue - m->pc) == PQ_FREE)) {
|
1998-07-11 07:46:16 +00:00
|
|
|
printf(
|
|
|
|
"vm_page_free: pindex(%lu), busy(%d), PG_BUSY(%d), hold(%d)\n",
|
|
|
|
(u_long)m->pindex, m->busy, (m->flags & PG_BUSY) ? 1 : 0,
|
|
|
|
m->hold_count);
|
1996-09-08 20:44:49 +00:00
|
|
|
if ((m->queue - m->pc) == PQ_FREE)
|
1996-06-05 03:31:49 +00:00
|
|
|
panic("vm_page_free: freeing free page");
|
|
|
|
else
|
|
|
|
panic("vm_page_free: freeing busy page");
|
|
|
|
}
|
|
|
|
|
1999-01-21 08:29:12 +00:00
|
|
|
/*
|
|
|
|
* unqueue, then remove page. Note that we cannot destroy
|
|
|
|
* the page here because we do not want to call the pager's
|
|
|
|
* callback routine until after we've put the page on the
|
|
|
|
* appropriate free queue.
|
|
|
|
*/
|
2001-07-04 23:27:09 +00:00
|
|
|
vm_pageq_remove_nowakeup(m);
|
1998-01-31 11:56:53 +00:00
|
|
|
vm_page_remove(m);
|
|
|
|
|
1999-01-21 08:29:12 +00:00
|
|
|
/*
|
|
|
|
* If fictitious remove object association and
|
|
|
|
* return, otherwise delay object association removal.
|
|
|
|
*/
|
1996-06-05 03:31:49 +00:00
|
|
|
if ((m->flags & PG_FICTITIOUS) != 0) {
|
1999-01-21 08:29:12 +00:00
|
|
|
splx(s);
|
|
|
|
return;
|
1996-06-05 03:31:49 +00:00
|
|
|
}
|
1998-01-31 11:56:53 +00:00
|
|
|
|
1998-03-01 04:18:54 +00:00
|
|
|
m->valid = 0;
|
1999-08-17 05:08:39 +00:00
|
|
|
vm_page_undirty(m);
|
1998-03-01 04:18:54 +00:00
|
|
|
|
1996-06-05 03:31:49 +00:00
|
|
|
if (m->wire_count != 0) {
|
|
|
|
if (m->wire_count > 1) {
|
1999-07-01 19:53:43 +00:00
|
|
|
panic("vm_page_free: invalid wire count (%d), pindex: 0x%lx",
|
|
|
|
m->wire_count, (long)m->pindex);
|
1996-06-05 03:31:49 +00:00
|
|
|
}
|
1999-11-10 05:23:19 +00:00
|
|
|
panic("vm_page_free: freeing wired page\n");
|
1996-06-05 03:31:49 +00:00
|
|
|
}
|
|
|
|
|
1999-01-21 08:29:12 +00:00
|
|
|
/*
|
|
|
|
* If we've exhausted the object's resident pages we want to free
|
|
|
|
* it up.
|
|
|
|
*/
|
|
|
|
if (object &&
|
|
|
|
(object->type == OBJT_VNODE) &&
|
|
|
|
((object->flags & OBJ_DEAD) == 0)
|
|
|
|
) {
|
|
|
|
struct vnode *vp = (struct vnode *)object->handle;
|
|
|
|
|
2002-08-04 10:29:36 +00:00
|
|
|
if (vp) {
|
|
|
|
VI_LOCK(vp);
|
|
|
|
if (VSHOULDFREE(vp))
|
|
|
|
vfree(vp);
|
|
|
|
VI_UNLOCK(vp);
|
|
|
|
}
|
1998-01-17 09:17:02 +00:00
|
|
|
}
|
|
|
|
|
2000-05-29 22:40:54 +00:00
|
|
|
/*
|
|
|
|
* Clear the UNMANAGED flag when freeing an unmanaged page.
|
|
|
|
*/
|
|
|
|
if (m->flags & PG_UNMANAGED) {
|
2002-03-10 21:52:48 +00:00
|
|
|
m->flags &= ~PG_UNMANAGED;
|
2000-05-29 22:40:54 +00:00
|
|
|
}
|
1998-07-26 18:15:20 +00:00
|
|
|
|
2002-02-19 23:19:30 +00:00
|
|
|
if (m->hold_count != 0) {
|
|
|
|
m->flags &= ~PG_ZERO;
|
|
|
|
m->queue = PQ_HOLD;
|
|
|
|
} else
|
|
|
|
m->queue = PQ_FREE + m->pc;
|
1996-09-28 03:33:40 +00:00
|
|
|
pq = &vm_page_queues[m->queue];
|
2002-07-04 22:07:37 +00:00
|
|
|
mtx_lock_spin(&vm_page_queue_free_mtx);
|
1999-07-31 18:31:00 +00:00
|
|
|
pq->lcnt++;
|
1996-09-28 03:33:40 +00:00
|
|
|
++(*pq->cnt);
|
1998-01-17 09:17:02 +00:00
|
|
|
|
1999-02-08 00:37:36 +00:00
|
|
|
/*
|
|
|
|
* Put zero'd pages on the end ( where we look for zero'd pages
|
|
|
|
* first ) and non-zerod pages at the head.
|
|
|
|
*/
|
|
|
|
if (m->flags & PG_ZERO) {
|
1999-10-30 07:37:14 +00:00
|
|
|
TAILQ_INSERT_TAIL(&pq->pl, m, pageq);
|
1999-01-21 08:29:12 +00:00
|
|
|
++vm_page_zero_count;
|
1999-02-08 00:37:36 +00:00
|
|
|
} else {
|
1999-10-30 07:37:14 +00:00
|
|
|
TAILQ_INSERT_HEAD(&pq->pl, m, pageq);
|
1996-06-05 03:31:49 +00:00
|
|
|
}
|
2002-07-04 22:07:37 +00:00
|
|
|
mtx_unlock_spin(&vm_page_queue_free_mtx);
|
1996-06-05 03:31:49 +00:00
|
|
|
vm_page_free_wakeup();
|
1996-06-16 20:37:31 +00:00
|
|
|
splx(s);
|
1996-06-05 03:31:49 +00:00
|
|
|
}
|
1994-05-25 09:21:21 +00:00
|
|
|
|
2000-05-29 22:40:54 +00:00
|
|
|
/*
|
|
|
|
* vm_page_unmanage:
|
|
|
|
*
|
|
|
|
* Prevent PV management from being done on the page. The page is
|
|
|
|
* removed from the paging queues as if it were wired, and as a
|
|
|
|
* consequence of no longer being managed the pageout daemon will not
|
|
|
|
* touch it (since there is no way to locate the pte mappings for the
|
|
|
|
* page). madvise() calls that mess with the pmap will also no longer
|
|
|
|
* operate on the page.
|
|
|
|
*
|
|
|
|
* Beyond that the page is still reasonably 'normal'. Freeing the page
|
|
|
|
* will clear the flag.
|
|
|
|
*
|
|
|
|
* This routine is used by OBJT_PHYS objects - objects using unswappable
|
|
|
|
* physical memory as backing store rather then swap-backed memory and
|
|
|
|
* will eventually be extended to support 4MB unmanaged physical
|
|
|
|
* mappings.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_unmanage(vm_page_t m)
|
|
|
|
{
|
|
|
|
int s;
|
|
|
|
|
|
|
|
s = splvm();
|
2002-07-13 23:55:30 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
2000-05-29 22:40:54 +00:00
|
|
|
if ((m->flags & PG_UNMANAGED) == 0) {
|
|
|
|
if (m->wire_count == 0)
|
2001-07-04 23:27:09 +00:00
|
|
|
vm_pageq_remove(m);
|
2000-05-29 22:40:54 +00:00
|
|
|
}
|
|
|
|
vm_page_flag_set(m, PG_UNMANAGED);
|
|
|
|
splx(s);
|
|
|
|
}
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* vm_page_wire:
|
|
|
|
*
|
|
|
|
* Mark this page as wired down by yet
|
|
|
|
* another map, removing it from paging queues
|
|
|
|
* as necessary.
|
|
|
|
*
|
|
|
|
* The page queues must be locked.
|
1998-12-23 01:52:47 +00:00
|
|
|
* This routine may not block.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1995-05-30 08:16:23 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_wire(vm_page_t m)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
1994-05-25 09:21:21 +00:00
|
|
|
int s;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2000-05-29 22:40:54 +00:00
|
|
|
/*
|
|
|
|
* Only bump the wire statistics if the page is not already wired,
|
|
|
|
* and only unqueue the page if it is on some queue (if it is unmanaged
|
|
|
|
* it is already off the queues).
|
|
|
|
*/
|
1998-11-11 15:07:57 +00:00
|
|
|
s = splvm();
|
2002-07-14 23:51:55 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
1996-01-19 04:00:31 +00:00
|
|
|
if (m->wire_count == 0) {
|
2000-05-29 22:40:54 +00:00
|
|
|
if ((m->flags & PG_UNMANAGED) == 0)
|
2001-07-04 23:27:09 +00:00
|
|
|
vm_pageq_remove(m);
|
2003-01-05 01:31:45 +00:00
|
|
|
atomic_add_int(&cnt.v_wire_count, 1);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
1996-01-19 04:00:31 +00:00
|
|
|
m->wire_count++;
|
2001-08-22 04:01:56 +00:00
|
|
|
KASSERT(m->wire_count != 0, ("vm_page_wire: wire_count overflow m=%p", m));
|
1998-11-11 15:07:57 +00:00
|
|
|
splx(s);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_unwire:
|
|
|
|
*
|
|
|
|
* Release one wiring of this page, potentially
|
|
|
|
* enabling it to be paged again.
|
|
|
|
*
|
1999-01-21 08:29:12 +00:00
|
|
|
* Many pages placed on the inactive queue should actually go
|
|
|
|
* into the cache, but it is difficult to figure out which. What
|
|
|
|
* we do instead, if the inactive target is well met, is to put
|
|
|
|
* clean pages at the head of the inactive queue instead of the tail.
|
|
|
|
* This will cause them to be moved to the cache more quickly and
|
|
|
|
* if not actively re-referenced, freed more quickly. If we just
|
|
|
|
* stick these pages at the end of the inactive queue, heavy filesystem
|
|
|
|
* meta-data accesses can cause an unnecessary paging load on memory bound
|
|
|
|
* processes. This optimization causes one-time-use metadata to be
|
|
|
|
* reused more quickly.
|
|
|
|
*
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
* BUT, if we are in a low-memory situation we have no choice but to
|
|
|
|
* put clean pages on the cache queue.
|
|
|
|
*
|
2000-03-26 15:20:23 +00:00
|
|
|
* A number of routines use vm_page_unwire() to guarantee that the page
|
1999-01-24 06:00:31 +00:00
|
|
|
* will go into either the inactive or active queues, and will NEVER
|
|
|
|
* be placed in the cache - for example, just after dirtying a page.
|
|
|
|
* dirty pages in the cache are not allowed.
|
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* The page queues must be locked.
|
1998-12-23 01:52:47 +00:00
|
|
|
* This routine may not block.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
1995-05-30 08:16:23 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_unwire(vm_page_t m, int activate)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
1994-05-25 09:21:21 +00:00
|
|
|
int s;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
1996-05-18 03:38:05 +00:00
|
|
|
s = splvm();
|
2002-07-13 20:55:21 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
1998-01-31 11:56:53 +00:00
|
|
|
if (m->wire_count > 0) {
|
1996-01-19 04:00:31 +00:00
|
|
|
m->wire_count--;
|
1998-01-31 11:56:53 +00:00
|
|
|
if (m->wire_count == 0) {
|
2003-01-05 01:31:45 +00:00
|
|
|
atomic_subtract_int(&cnt.v_wire_count, 1);
|
2000-05-29 22:40:54 +00:00
|
|
|
if (m->flags & PG_UNMANAGED) {
|
|
|
|
;
|
2002-03-04 18:55:26 +00:00
|
|
|
} else if (activate)
|
|
|
|
vm_pageq_enqueue(PQ_ACTIVE, m);
|
|
|
|
else {
|
2000-12-26 19:41:38 +00:00
|
|
|
vm_page_flag_clear(m, PG_WINATCFLS);
|
2002-03-04 18:55:26 +00:00
|
|
|
vm_pageq_enqueue(PQ_INACTIVE, m);
|
1998-10-28 13:37:02 +00:00
|
|
|
}
|
1998-01-31 11:56:53 +00:00
|
|
|
}
|
|
|
|
} else {
|
|
|
|
panic("vm_page_unwire: invalid wire count: %d\n", m->wire_count);
|
1995-03-25 08:47:35 +00:00
|
|
|
}
|
|
|
|
splx(s);
|
|
|
|
}
|
|
|
|
|
1996-05-18 03:38:05 +00:00
|
|
|
|
1994-05-25 09:21:21 +00:00
|
|
|
/*
|
1999-01-21 08:29:12 +00:00
|
|
|
* Move the specified page to the inactive queue. If the page has
|
|
|
|
* any associated swap, the swap is deallocated.
|
1998-12-23 01:52:47 +00:00
|
|
|
*
|
1999-09-17 04:56:40 +00:00
|
|
|
* Normally athead is 0 resulting in LRU operation. athead is set
|
|
|
|
* to 1 if we want this page to be 'as if it were placed in the cache',
|
|
|
|
* except without unmapping it from the process address space.
|
|
|
|
*
|
1998-12-23 01:52:47 +00:00
|
|
|
* This routine may not block.
|
1994-05-25 09:21:21 +00:00
|
|
|
*/
|
1999-09-17 04:56:40 +00:00
|
|
|
static __inline void
|
|
|
|
_vm_page_deactivate(vm_page_t m, int athead)
|
1994-05-25 09:21:21 +00:00
|
|
|
{
|
1996-05-31 00:38:04 +00:00
|
|
|
int s;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
2002-08-02 05:23:58 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
1994-05-25 09:21:21 +00:00
|
|
|
/*
|
1998-10-28 13:41:43 +00:00
|
|
|
* Ignore if already inactive.
|
1994-05-25 09:21:21 +00:00
|
|
|
*/
|
1996-01-19 04:00:31 +00:00
|
|
|
if (m->queue == PQ_INACTIVE)
|
|
|
|
return;
|
1994-05-25 09:21:21 +00:00
|
|
|
|
1996-05-31 00:38:04 +00:00
|
|
|
s = splvm();
|
2000-05-29 22:40:54 +00:00
|
|
|
if (m->wire_count == 0 && (m->flags & PG_UNMANAGED) == 0) {
|
1996-09-08 20:44:49 +00:00
|
|
|
if ((m->queue - m->pc) == PQ_CACHE)
|
1995-03-25 08:47:35 +00:00
|
|
|
cnt.v_reactivated++;
|
2000-12-26 19:41:38 +00:00
|
|
|
vm_page_flag_clear(m, PG_WINATCFLS);
|
2001-07-04 23:27:09 +00:00
|
|
|
vm_pageq_remove(m);
|
1999-09-17 04:56:40 +00:00
|
|
|
if (athead)
|
1999-10-30 07:37:14 +00:00
|
|
|
TAILQ_INSERT_HEAD(&vm_page_queues[PQ_INACTIVE].pl, m, pageq);
|
1999-09-17 04:56:40 +00:00
|
|
|
else
|
1999-10-30 07:37:14 +00:00
|
|
|
TAILQ_INSERT_TAIL(&vm_page_queues[PQ_INACTIVE].pl, m, pageq);
|
1996-01-19 04:00:31 +00:00
|
|
|
m->queue = PQ_INACTIVE;
|
1999-07-31 18:31:00 +00:00
|
|
|
vm_page_queues[PQ_INACTIVE].lcnt++;
|
1994-05-25 09:21:21 +00:00
|
|
|
cnt.v_inactive_count++;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
}
|
1996-05-31 00:38:04 +00:00
|
|
|
splx(s);
|
1994-05-25 09:21:21 +00:00
|
|
|
}
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
1999-09-17 04:56:40 +00:00
|
|
|
void
|
|
|
|
vm_page_deactivate(vm_page_t m)
|
|
|
|
{
|
|
|
|
_vm_page_deactivate(m, 0);
|
|
|
|
}
|
|
|
|
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
/*
|
|
|
|
* vm_page_try_to_cache:
|
|
|
|
*
|
|
|
|
* Returns 0 on failure, 1 on success
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
vm_page_try_to_cache(vm_page_t m)
|
|
|
|
{
|
2001-05-19 01:28:09 +00:00
|
|
|
|
2002-07-20 20:58:46 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
if (m->dirty || m->hold_count || m->busy || m->wire_count ||
|
|
|
|
(m->flags & (PG_BUSY|PG_UNMANAGED))) {
|
2002-03-10 21:52:48 +00:00
|
|
|
return (0);
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
}
|
|
|
|
vm_page_test_dirty(m);
|
|
|
|
if (m->dirty)
|
2002-03-10 21:52:48 +00:00
|
|
|
return (0);
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
vm_page_cache(m);
|
2002-03-10 21:52:48 +00:00
|
|
|
return (1);
|
Implement a low-memory deadlock solution.
Removed most of the hacks that were trying to deal with low-memory
situations prior to now.
The new code is based on the concept that I/O must be able to function in
a low memory situation. All major modules related to I/O (except
networking) have been adjusted to allow allocation out of the system
reserve memory pool. These modules now detect a low memory situation but
rather then block they instead continue to operate, then return resources
to the memory pool instead of cache them or leave them wired.
Code has been added to stall in a low-memory situation prior to a vnode
being locked.
Thus situations where a process blocks in a low-memory condition while
holding a locked vnode have been reduced to near nothing. Not only will
I/O continue to operate, but many prior deadlock conditions simply no
longer exist.
Implement a number of VFS/BIO fixes
(found by Ian): in biodone(), bogus-page replacement code, the loop
was not properly incrementing loop variables prior to a continue
statement. We do not believe this code can be hit anyway but we
aren't taking any chances. We'll turn the whole section into a
panic (as it already is in brelse()) after the release is rolled.
In biodone(), the foff calculation was incorrectly
clamped to the iosize, causing the wrong foff to be calculated
for pages in the case of an I/O error or biodone() called without
initiating I/O. The problem always caused a panic before. Now it
doesn't. The problem is mainly an issue with NFS.
Fixed casts for ~PAGE_MASK. This code worked properly before only
because the calculations use signed arithmatic. Better to properly
extend PAGE_MASK first before inverting it for the 64 bit masking
op.
In brelse(), the bogus_page fixup code was improperly throwing
away the original contents of 'm' when it did the j-loop to
fix the bogus pages. The result was that it would potentially
invalidate parts of the *WRONG* page(!), leading to corruption.
There may still be cases where a background bitmap write is
being duplicated, causing potential corruption. We have identified
a potentially serious bug related to this but the fix is still TBD.
So instead this patch contains a KASSERT to detect the problem
and panic the machine rather then continue to corrupt the filesystem.
The problem does not occur very often.. it is very hard to
reproduce, and it may or may not be the cause of the corruption
people have reported.
Review by: (VFS/BIO: mckusick, Ian Dowse <iedowse@maths.tcd.ie>)
Testing by: (VM/Deadlock) Paul Saab <ps@yahoo-inc.com>
2000-11-18 23:06:26 +00:00
|
|
|
}
|
|
|
|
|
2001-05-24 07:22:27 +00:00
|
|
|
/*
|
|
|
|
* vm_page_try_to_free()
|
|
|
|
*
|
|
|
|
* Attempt to free the page. If we cannot free it, we do nothing.
|
|
|
|
* 1 is returned on success, 0 on failure.
|
|
|
|
*/
|
|
|
|
int
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_try_to_free(vm_page_t m)
|
2001-05-24 07:22:27 +00:00
|
|
|
{
|
2002-07-20 20:12:57 +00:00
|
|
|
|
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
2003-06-19 01:50:14 +00:00
|
|
|
if (m->object != NULL)
|
|
|
|
VM_OBJECT_LOCK_ASSERT(m->object, MA_OWNED);
|
2001-05-24 07:22:27 +00:00
|
|
|
if (m->dirty || m->hold_count || m->busy || m->wire_count ||
|
|
|
|
(m->flags & (PG_BUSY|PG_UNMANAGED))) {
|
2002-03-10 21:52:48 +00:00
|
|
|
return (0);
|
2001-05-24 07:22:27 +00:00
|
|
|
}
|
|
|
|
vm_page_test_dirty(m);
|
|
|
|
if (m->dirty)
|
2002-03-10 21:52:48 +00:00
|
|
|
return (0);
|
2001-05-24 07:22:27 +00:00
|
|
|
vm_page_busy(m);
|
2002-11-16 07:44:25 +00:00
|
|
|
pmap_remove_all(m);
|
2001-05-24 07:22:27 +00:00
|
|
|
vm_page_free(m);
|
2002-03-10 21:52:48 +00:00
|
|
|
return (1);
|
2001-05-24 07:22:27 +00:00
|
|
|
}
|
|
|
|
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
/*
|
|
|
|
* vm_page_cache
|
|
|
|
*
|
1999-01-21 08:29:12 +00:00
|
|
|
* Put the specified page onto the page cache queue (if appropriate).
|
|
|
|
*
|
1998-12-23 01:52:47 +00:00
|
|
|
* This routine may not block.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*/
|
1995-05-30 08:16:23 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_cache(vm_page_t m)
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
{
|
|
|
|
int s;
|
|
|
|
|
2002-07-20 19:34:21 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
2000-05-29 22:40:54 +00:00
|
|
|
if ((m->flags & (PG_BUSY|PG_UNMANAGED)) || m->busy || m->wire_count) {
|
1996-03-28 04:53:28 +00:00
|
|
|
printf("vm_page_cache: attempting to cache busy page\n");
|
1996-01-19 04:00:31 +00:00
|
|
|
return;
|
1996-03-28 04:53:28 +00:00
|
|
|
}
|
1996-09-08 20:44:49 +00:00
|
|
|
if ((m->queue - m->pc) == PQ_CACHE)
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
return;
|
|
|
|
|
1999-01-24 01:04:04 +00:00
|
|
|
/*
|
|
|
|
* Remove all pmaps and indicate that the page is not
|
1999-01-24 02:29:26 +00:00
|
|
|
* writeable or mapped.
|
1999-01-24 01:04:04 +00:00
|
|
|
*/
|
2002-11-16 07:44:25 +00:00
|
|
|
pmap_remove_all(m);
|
1996-05-24 05:20:15 +00:00
|
|
|
if (m->dirty != 0) {
|
1999-07-01 19:53:43 +00:00
|
|
|
panic("vm_page_cache: caching a dirty page, pindex: %ld",
|
|
|
|
(long)m->pindex);
|
1996-05-24 05:20:15 +00:00
|
|
|
}
|
1996-05-18 03:38:05 +00:00
|
|
|
s = splvm();
|
2001-07-04 23:27:09 +00:00
|
|
|
vm_pageq_remove_nowakeup(m);
|
2002-03-04 18:55:26 +00:00
|
|
|
vm_pageq_enqueue(PQ_CACHE + m->pc, m);
|
1996-06-16 20:37:31 +00:00
|
|
|
vm_page_free_wakeup();
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
splx(s);
|
|
|
|
}
|
|
|
|
|
1999-09-17 04:56:40 +00:00
|
|
|
/*
|
|
|
|
* vm_page_dontneed
|
|
|
|
*
|
|
|
|
* Cache, deactivate, or do nothing as appropriate. This routine
|
|
|
|
* is typically used by madvise() MADV_DONTNEED.
|
|
|
|
*
|
|
|
|
* Generally speaking we want to move the page into the cache so
|
|
|
|
* it gets reused quickly. However, this can result in a silly syndrome
|
|
|
|
* due to the page recycling too quickly. Small objects will not be
|
|
|
|
* fully cached. On the otherhand, if we move the page to the inactive
|
|
|
|
* queue we wind up with a problem whereby very large objects
|
|
|
|
* unnecessarily blow away our inactive and cache queues.
|
|
|
|
*
|
|
|
|
* The solution is to move the pages based on a fixed weighting. We
|
|
|
|
* either leave them alone, deactivate them, or move them to the cache,
|
|
|
|
* where moving them to the cache has the highest weighting.
|
|
|
|
* By forcing some pages into other queues we eventually force the
|
|
|
|
* system to balance the queues, potentially recovering other unrelated
|
|
|
|
* space from active. The idea is to not force this to happen too
|
|
|
|
* often.
|
|
|
|
*/
|
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_dontneed(vm_page_t m)
|
1999-09-17 04:56:40 +00:00
|
|
|
{
|
|
|
|
static int dnweight;
|
|
|
|
int dnw;
|
|
|
|
int head;
|
|
|
|
|
2002-07-23 04:39:48 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
1999-09-17 04:56:40 +00:00
|
|
|
dnw = ++dnweight;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* occassionally leave the page alone
|
|
|
|
*/
|
|
|
|
if ((dnw & 0x01F0) == 0 ||
|
|
|
|
m->queue == PQ_INACTIVE ||
|
|
|
|
m->queue - m->pc == PQ_CACHE
|
|
|
|
) {
|
|
|
|
if (m->act_count >= ACT_INIT)
|
|
|
|
--m->act_count;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (m->dirty == 0)
|
|
|
|
vm_page_test_dirty(m);
|
|
|
|
|
|
|
|
if (m->dirty || (dnw & 0x0070) == 0) {
|
|
|
|
/*
|
|
|
|
* Deactivate the page 3 times out of 32.
|
|
|
|
*/
|
|
|
|
head = 0;
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* Cache the page 28 times out of every 32. Note that
|
|
|
|
* the page is deactivated instead of cached, but placed
|
|
|
|
* at the head of the queue instead of the tail.
|
|
|
|
*/
|
|
|
|
head = 1;
|
|
|
|
}
|
|
|
|
_vm_page_deactivate(m, head);
|
|
|
|
}
|
|
|
|
|
1998-02-05 03:32:49 +00:00
|
|
|
/*
|
|
|
|
* Grab a page, waiting until we are waken up due to the page
|
|
|
|
* changing state. We keep on waiting, if the page continues
|
|
|
|
* to be in the object. If the page doesn't exist, allocate it.
|
1998-12-23 01:52:47 +00:00
|
|
|
*
|
|
|
|
* This routine may block.
|
1998-02-05 03:32:49 +00:00
|
|
|
*/
|
|
|
|
vm_page_t
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_grab(vm_object_t object, vm_pindex_t pindex, int allocflags)
|
1998-02-05 03:32:49 +00:00
|
|
|
{
|
|
|
|
vm_page_t m;
|
2003-06-07 23:22:04 +00:00
|
|
|
int s, generation, is_object_locked;
|
1998-02-05 03:32:49 +00:00
|
|
|
|
2003-06-07 23:22:04 +00:00
|
|
|
/*
|
|
|
|
* Remove is_object_locked after vm_object locking is finished.
|
|
|
|
*/
|
|
|
|
if (!(is_object_locked = VM_OBJECT_LOCKED(object)))
|
|
|
|
GIANT_REQUIRED;
|
1998-02-05 03:32:49 +00:00
|
|
|
retrylookup:
|
|
|
|
if ((m = vm_page_lookup(object, pindex)) != NULL) {
|
2002-08-04 19:05:20 +00:00
|
|
|
vm_page_lock_queues();
|
1998-02-05 03:32:49 +00:00
|
|
|
if (m->busy || (m->flags & PG_BUSY)) {
|
|
|
|
generation = object->generation;
|
|
|
|
|
|
|
|
s = splvm();
|
|
|
|
while ((object->generation == generation) &&
|
|
|
|
(m->busy || (m->flags & PG_BUSY))) {
|
1998-09-04 08:06:57 +00:00
|
|
|
vm_page_flag_set(m, PG_WANTED | PG_REFERENCED);
|
2003-06-07 23:22:04 +00:00
|
|
|
if (is_object_locked)
|
|
|
|
VM_OBJECT_UNLOCK(object);
|
|
|
|
msleep(m, &vm_page_queue_mtx, PDROP | PVM, "pgrbwt", 0);
|
|
|
|
if (is_object_locked)
|
|
|
|
VM_OBJECT_LOCK(object);
|
1998-02-05 03:32:49 +00:00
|
|
|
if ((allocflags & VM_ALLOC_RETRY) == 0) {
|
|
|
|
splx(s);
|
|
|
|
return NULL;
|
|
|
|
}
|
2003-06-07 23:22:04 +00:00
|
|
|
vm_page_lock_queues();
|
1998-02-05 03:32:49 +00:00
|
|
|
}
|
2002-08-04 19:05:20 +00:00
|
|
|
vm_page_unlock_queues();
|
1998-02-05 03:32:49 +00:00
|
|
|
splx(s);
|
|
|
|
goto retrylookup;
|
|
|
|
} else {
|
2002-07-28 23:46:19 +00:00
|
|
|
if (allocflags & VM_ALLOC_WIRED)
|
|
|
|
vm_page_wire(m);
|
1998-09-04 08:06:57 +00:00
|
|
|
vm_page_busy(m);
|
2002-07-28 23:46:19 +00:00
|
|
|
vm_page_unlock_queues();
|
1998-02-05 03:32:49 +00:00
|
|
|
return m;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
m = vm_page_alloc(object, pindex, allocflags & ~VM_ALLOC_RETRY);
|
|
|
|
if (m == NULL) {
|
2003-06-07 23:22:04 +00:00
|
|
|
if (is_object_locked)
|
|
|
|
VM_OBJECT_UNLOCK(object);
|
1998-02-05 03:32:49 +00:00
|
|
|
VM_WAIT;
|
2003-06-07 23:22:04 +00:00
|
|
|
if (is_object_locked)
|
|
|
|
VM_OBJECT_LOCK(object);
|
1998-02-05 03:32:49 +00:00
|
|
|
if ((allocflags & VM_ALLOC_RETRY) == 0)
|
|
|
|
return NULL;
|
|
|
|
goto retrylookup;
|
|
|
|
}
|
|
|
|
|
|
|
|
return m;
|
|
|
|
}
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
|
|
|
/*
|
1999-04-05 19:38:30 +00:00
|
|
|
* Mapping function for valid bits or for dirty bits in
|
1998-12-23 01:52:47 +00:00
|
|
|
* a page. May not block.
|
1999-04-05 19:38:30 +00:00
|
|
|
*
|
|
|
|
* Inputs are required to range within a page.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*/
|
1998-04-15 17:47:40 +00:00
|
|
|
__inline int
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
vm_page_bits(int base, int size)
|
|
|
|
{
|
1999-04-05 19:38:30 +00:00
|
|
|
int first_bit;
|
|
|
|
int last_bit;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
1999-04-05 19:38:30 +00:00
|
|
|
KASSERT(
|
|
|
|
base + size <= PAGE_SIZE,
|
|
|
|
("vm_page_bits: illegal base/size %d/%d", base, size)
|
|
|
|
);
|
Some VM improvements, including elimination of alot of Sig-11
problems. Tor Egge and others have helped with various VM bugs
lately, but don't blame him -- blame me!!!
pmap.c:
1) Create an object for kernel page table allocations. This
fixes a bogus allocation method previously used for such, by
grabbing pages from the kernel object, using bogus pindexes.
(This was a code cleanup, and perhaps a minor system stability
issue.)
pmap.c:
2) Pre-set the modify and accessed bits when prudent. This will
decrease bus traffic under certain circumstances.
vfs_bio.c, vfs_cluster.c:
3) Rather than calculating the beginning virtual byte offset
multiple times, stick the offset into the buffer header, so
that the calculated offset can be reused. (Long long multiplies
are often expensive, and this is a probably unmeasurable performance
improvement, and code cleanup.)
vfs_bio.c:
4) Handle write recursion more intelligently (but not perfectly) so
that it is less likely to cause a system panic, and is also
much more robust.
vfs_bio.c:
5) getblk incorrectly wrote out blocks that are incorrectly sized.
The problem is fixed, and writes blocks out ONLY when B_DELWRI
is true.
vfs_bio.c:
6) Check that already constituted buffers have fully valid pages. If
not, then make sure that the B_CACHE bit is not set. (This was
a major source of Sig-11 type problems.)
vfs_bio.c:
7) Fix a potential system deadlock due to an incorrectly specified
sleep priority while waiting for a buffer write operation. The
change that I made opens the system up to serious problems, and
we need to examine the issue of process sleep priorities.
vfs_cluster.c, vfs_bio.c:
8) Make clustered reads work more correctly (and more completely)
when buffers are already constituted, but not fully valid.
(This was another system reliability issue.)
vfs_subr.c, ffs_inode.c:
9) Create a vtruncbuf function, which is used by filesystems that
can truncate files. The vinvalbuf forced a file sync type operation,
while vtruncbuf only invalidates the buffers past the new end of file,
and also invalidates the appropriate pages. (This was a system reliabiliy
and performance issue.)
10) Modify FFS to use vtruncbuf.
vm_object.c:
11) Make the object rundown mechanism for OBJT_VNODE type objects work
more correctly. Included in that fix, create pager entries for
the OBJT_DEAD pager type, so that paging requests that might slip
in during race conditions are properly handled. (This was a system
reliability issue.)
vm_page.c:
12) Make some of the page validation routines be a little less picky
about arguments passed to them. Also, support page invalidation
change the object generation count so that we handle generation
counts a little more robustly.
vm_pageout.c:
13) Further reduce pageout daemon activity when the system doesn't
need help from it. There should be no additional performance
decrease even when the pageout daemon is running. (This was
a significant performance issue.)
vnode_pager.c:
14) Teach the vnode pager to handle race conditions during vnode
deallocations.
1998-03-16 01:56:03 +00:00
|
|
|
|
1999-04-05 19:38:30 +00:00
|
|
|
if (size == 0) /* handle degenerate case */
|
2002-03-10 21:52:48 +00:00
|
|
|
return (0);
|
Some VM improvements, including elimination of alot of Sig-11
problems. Tor Egge and others have helped with various VM bugs
lately, but don't blame him -- blame me!!!
pmap.c:
1) Create an object for kernel page table allocations. This
fixes a bogus allocation method previously used for such, by
grabbing pages from the kernel object, using bogus pindexes.
(This was a code cleanup, and perhaps a minor system stability
issue.)
pmap.c:
2) Pre-set the modify and accessed bits when prudent. This will
decrease bus traffic under certain circumstances.
vfs_bio.c, vfs_cluster.c:
3) Rather than calculating the beginning virtual byte offset
multiple times, stick the offset into the buffer header, so
that the calculated offset can be reused. (Long long multiplies
are often expensive, and this is a probably unmeasurable performance
improvement, and code cleanup.)
vfs_bio.c:
4) Handle write recursion more intelligently (but not perfectly) so
that it is less likely to cause a system panic, and is also
much more robust.
vfs_bio.c:
5) getblk incorrectly wrote out blocks that are incorrectly sized.
The problem is fixed, and writes blocks out ONLY when B_DELWRI
is true.
vfs_bio.c:
6) Check that already constituted buffers have fully valid pages. If
not, then make sure that the B_CACHE bit is not set. (This was
a major source of Sig-11 type problems.)
vfs_bio.c:
7) Fix a potential system deadlock due to an incorrectly specified
sleep priority while waiting for a buffer write operation. The
change that I made opens the system up to serious problems, and
we need to examine the issue of process sleep priorities.
vfs_cluster.c, vfs_bio.c:
8) Make clustered reads work more correctly (and more completely)
when buffers are already constituted, but not fully valid.
(This was another system reliability issue.)
vfs_subr.c, ffs_inode.c:
9) Create a vtruncbuf function, which is used by filesystems that
can truncate files. The vinvalbuf forced a file sync type operation,
while vtruncbuf only invalidates the buffers past the new end of file,
and also invalidates the appropriate pages. (This was a system reliabiliy
and performance issue.)
10) Modify FFS to use vtruncbuf.
vm_object.c:
11) Make the object rundown mechanism for OBJT_VNODE type objects work
more correctly. Included in that fix, create pager entries for
the OBJT_DEAD pager type, so that paging requests that might slip
in during race conditions are properly handled. (This was a system
reliability issue.)
vm_page.c:
12) Make some of the page validation routines be a little less picky
about arguments passed to them. Also, support page invalidation
change the object generation count so that we handle generation
counts a little more robustly.
vm_pageout.c:
13) Further reduce pageout daemon activity when the system doesn't
need help from it. There should be no additional performance
decrease even when the pageout daemon is running. (This was
a significant performance issue.)
vnode_pager.c:
14) Teach the vnode pager to handle race conditions during vnode
deallocations.
1998-03-16 01:56:03 +00:00
|
|
|
|
1999-04-05 19:38:30 +00:00
|
|
|
first_bit = base >> DEV_BSHIFT;
|
|
|
|
last_bit = (base + size - 1) >> DEV_BSHIFT;
|
|
|
|
|
|
|
|
return ((2 << last_bit) - (1 << first_bit));
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
}
|
|
|
|
|
1995-09-03 19:57:25 +00:00
|
|
|
/*
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
* vm_page_set_validclean:
|
1999-04-05 19:38:30 +00:00
|
|
|
*
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
* Sets portions of a page valid and clean. The arguments are expected
|
|
|
|
* to be DEV_BSIZE aligned but if they aren't the bitmap is inclusive
|
|
|
|
* of any partial chunks touched by the range. The invalid portion of
|
|
|
|
* such chunks will be zero'd.
|
1999-04-05 19:38:30 +00:00
|
|
|
*
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
* This routine may not block.
|
|
|
|
*
|
|
|
|
* (base + size) must be less then or equal to PAGE_SIZE.
|
1995-09-03 19:57:25 +00:00
|
|
|
*/
|
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_set_validclean(vm_page_t m, int base, int size)
|
1995-09-03 19:57:25 +00:00
|
|
|
{
|
1999-04-05 19:38:30 +00:00
|
|
|
int pagebits;
|
|
|
|
int frag;
|
|
|
|
int endoff;
|
|
|
|
|
2002-11-23 19:10:31 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
1999-04-05 19:38:30 +00:00
|
|
|
if (size == 0) /* handle degenerate case */
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the base is not DEV_BSIZE aligned and the valid
|
|
|
|
* bit is clear, we have to zero out a portion of the
|
|
|
|
* first block.
|
|
|
|
*/
|
|
|
|
if ((frag = base & ~(DEV_BSIZE - 1)) != base &&
|
2002-04-15 16:00:03 +00:00
|
|
|
(m->valid & (1 << (base >> DEV_BSHIFT))) == 0)
|
|
|
|
pmap_zero_page_area(m, frag, base - frag);
|
1999-04-05 19:38:30 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If the ending offset is not DEV_BSIZE aligned and the
|
|
|
|
* valid bit is clear, we have to zero out a portion of
|
|
|
|
* the last block.
|
|
|
|
*/
|
|
|
|
endoff = base + size;
|
|
|
|
if ((frag = endoff & ~(DEV_BSIZE - 1)) != endoff &&
|
2002-04-15 16:00:03 +00:00
|
|
|
(m->valid & (1 << (endoff >> DEV_BSHIFT))) == 0)
|
|
|
|
pmap_zero_page_area(m, endoff,
|
|
|
|
DEV_BSIZE - (endoff & (DEV_BSIZE - 1)));
|
1999-04-05 19:38:30 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Set valid, clear dirty bits. If validating the entire
|
1999-12-12 03:19:33 +00:00
|
|
|
* page we can safely clear the pmap modify bit. We also
|
|
|
|
* use this opportunity to clear the PG_NOSYNC flag. If a process
|
|
|
|
* takes a write fault on a MAP_NOSYNC memory area the flag will
|
|
|
|
* be set again.
|
2001-12-14 01:16:57 +00:00
|
|
|
*
|
|
|
|
* We set valid bits inclusive of any overlap, but we can only
|
|
|
|
* clear dirty bits for DEV_BSIZE chunks that are fully within
|
|
|
|
* the range.
|
1999-04-05 19:38:30 +00:00
|
|
|
*/
|
|
|
|
pagebits = vm_page_bits(base, size);
|
1995-09-03 19:57:25 +00:00
|
|
|
m->valid |= pagebits;
|
2001-12-14 01:16:57 +00:00
|
|
|
#if 0 /* NOT YET */
|
|
|
|
if ((frag = base & (DEV_BSIZE - 1)) != 0) {
|
|
|
|
frag = DEV_BSIZE - frag;
|
|
|
|
base += frag;
|
|
|
|
size -= frag;
|
|
|
|
if (size < 0)
|
|
|
|
size = 0;
|
|
|
|
}
|
|
|
|
pagebits = vm_page_bits(base, size & (DEV_BSIZE - 1));
|
|
|
|
#endif
|
1995-09-03 19:57:25 +00:00
|
|
|
m->dirty &= ~pagebits;
|
1999-12-12 03:19:33 +00:00
|
|
|
if (base == 0 && size == PAGE_SIZE) {
|
2000-05-21 12:50:18 +00:00
|
|
|
pmap_clear_modify(m);
|
1999-12-12 03:19:33 +00:00
|
|
|
vm_page_flag_clear(m, PG_NOSYNC);
|
|
|
|
}
|
1995-09-03 19:57:25 +00:00
|
|
|
}
|
|
|
|
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
#if 0
|
|
|
|
|
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_set_dirty(vm_page_t m, int base, int size)
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
{
|
|
|
|
m->dirty |= vm_page_bits(base, size);
|
|
|
|
}
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_clear_dirty(vm_page_t m, int base, int size)
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
{
|
2001-07-04 16:20:28 +00:00
|
|
|
GIANT_REQUIRED;
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
m->dirty &= ~vm_page_bits(base, size);
|
|
|
|
}
|
|
|
|
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
/*
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
* vm_page_set_invalid:
|
|
|
|
*
|
|
|
|
* Invalidates DEV_BSIZE'd chunks within a page. Both the
|
|
|
|
* valid and dirty bits for the effected areas are cleared.
|
|
|
|
*
|
|
|
|
* May not block.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*/
|
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_set_invalid(vm_page_t m, int base, int size)
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
{
|
|
|
|
int bits;
|
|
|
|
|
2001-07-04 16:20:28 +00:00
|
|
|
GIANT_REQUIRED;
|
The VFS/BIO subsystem contained a number of hacks in order to optimize
piecemeal, middle-of-file writes for NFS. These hacks have caused no
end of trouble, especially when combined with mmap(). I've removed
them. Instead, NFS will issue a read-before-write to fully
instantiate the struct buf containing the write. NFS does, however,
optimize piecemeal appends to files. For most common file operations,
you will not notice the difference. The sole remaining fragment in
the VFS/BIO system is b_dirtyoff/end, which NFS uses to avoid cache
coherency issues with read-merge-write style operations. NFS also
optimizes the write-covers-entire-buffer case by avoiding the
read-before-write. There is quite a bit of room for further
optimization in these areas.
The VM system marks pages fully-valid (AKA vm_page_t->valid =
VM_PAGE_BITS_ALL) in several places, most noteably in vm_fault. This
is not correct operation. The vm_pager_get_pages() code is now
responsible for marking VM pages all-valid. A number of VM helper
routines have been added to aid in zeroing-out the invalid portions of
a VM page prior to the page being marked all-valid. This operation is
necessary to properly support mmap(). The zeroing occurs most often
when dealing with file-EOF situations. Several bugs have been fixed
in the NFS subsystem, including bits handling file and directory EOF
situations and buf->b_flags consistancy issues relating to clearing
B_ERROR & B_INVAL, and handling B_DONE.
getblk() and allocbuf() have been rewritten. B_CACHE operation is now
formally defined in comments and more straightforward in
implementation. B_CACHE for VMIO buffers is based on the validity of
the backing store. B_CACHE for non-VMIO buffers is based simply on
whether the buffer is B_INVAL or not (B_CACHE set if B_INVAL clear,
and vise-versa). biodone() is now responsible for setting B_CACHE
when a successful read completes. B_CACHE is also set when a bdwrite()
is initiated and when a bwrite() is initiated. VFS VOP_BWRITE
routines (there are only two - nfs_bwrite() and bwrite()) are now
expected to set B_CACHE. This means that bowrite() and bawrite() also
set B_CACHE indirectly.
There are a number of places in the code which were previously using
buf->b_bufsize (which is DEV_BSIZE aligned) when they should have
been using buf->b_bcount. These have been fixed. getblk() now clears
B_DONE on return because the rest of the system is so bad about
dealing with B_DONE.
Major fixes to NFS/TCP have been made. A server-side bug could cause
requests to be lost by the server due to nfs_realign() overwriting
other rpc's in the same TCP mbuf chain. The server's kernel must be
recompiled to get the benefit of the fixes.
Submitted by: Matthew Dillon <dillon@apollo.backplane.com>
1999-05-02 23:57:16 +00:00
|
|
|
bits = vm_page_bits(base, size);
|
|
|
|
m->valid &= ~bits;
|
|
|
|
m->dirty &= ~bits;
|
Some VM improvements, including elimination of alot of Sig-11
problems. Tor Egge and others have helped with various VM bugs
lately, but don't blame him -- blame me!!!
pmap.c:
1) Create an object for kernel page table allocations. This
fixes a bogus allocation method previously used for such, by
grabbing pages from the kernel object, using bogus pindexes.
(This was a code cleanup, and perhaps a minor system stability
issue.)
pmap.c:
2) Pre-set the modify and accessed bits when prudent. This will
decrease bus traffic under certain circumstances.
vfs_bio.c, vfs_cluster.c:
3) Rather than calculating the beginning virtual byte offset
multiple times, stick the offset into the buffer header, so
that the calculated offset can be reused. (Long long multiplies
are often expensive, and this is a probably unmeasurable performance
improvement, and code cleanup.)
vfs_bio.c:
4) Handle write recursion more intelligently (but not perfectly) so
that it is less likely to cause a system panic, and is also
much more robust.
vfs_bio.c:
5) getblk incorrectly wrote out blocks that are incorrectly sized.
The problem is fixed, and writes blocks out ONLY when B_DELWRI
is true.
vfs_bio.c:
6) Check that already constituted buffers have fully valid pages. If
not, then make sure that the B_CACHE bit is not set. (This was
a major source of Sig-11 type problems.)
vfs_bio.c:
7) Fix a potential system deadlock due to an incorrectly specified
sleep priority while waiting for a buffer write operation. The
change that I made opens the system up to serious problems, and
we need to examine the issue of process sleep priorities.
vfs_cluster.c, vfs_bio.c:
8) Make clustered reads work more correctly (and more completely)
when buffers are already constituted, but not fully valid.
(This was another system reliability issue.)
vfs_subr.c, ffs_inode.c:
9) Create a vtruncbuf function, which is used by filesystems that
can truncate files. The vinvalbuf forced a file sync type operation,
while vtruncbuf only invalidates the buffers past the new end of file,
and also invalidates the appropriate pages. (This was a system reliabiliy
and performance issue.)
10) Modify FFS to use vtruncbuf.
vm_object.c:
11) Make the object rundown mechanism for OBJT_VNODE type objects work
more correctly. Included in that fix, create pager entries for
the OBJT_DEAD pager type, so that paging requests that might slip
in during race conditions are properly handled. (This was a system
reliability issue.)
vm_page.c:
12) Make some of the page validation routines be a little less picky
about arguments passed to them. Also, support page invalidation
change the object generation count so that we handle generation
counts a little more robustly.
vm_pageout.c:
13) Further reduce pageout daemon activity when the system doesn't
need help from it. There should be no additional performance
decrease even when the pageout daemon is running. (This was
a significant performance issue.)
vnode_pager.c:
14) Teach the vnode pager to handle race conditions during vnode
deallocations.
1998-03-16 01:56:03 +00:00
|
|
|
m->object->generation++;
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
1999-04-05 19:38:30 +00:00
|
|
|
* vm_page_zero_invalid()
|
|
|
|
*
|
|
|
|
* The kernel assumes that the invalid portions of a page contain
|
|
|
|
* garbage, but such pages can be mapped into memory by user code.
|
|
|
|
* When this occurs, we must zero out the non-valid portions of the
|
|
|
|
* page so user code sees what it expects.
|
|
|
|
*
|
|
|
|
* Pages are most often semi-valid when the end of a file is mapped
|
|
|
|
* into memory and the file's size is not page aligned.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_page_zero_invalid(vm_page_t m, boolean_t setvalid)
|
|
|
|
{
|
|
|
|
int b;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Scan the valid bits looking for invalid sections that
|
|
|
|
* must be zerod. Invalid sub-DEV_BSIZE'd areas ( where the
|
|
|
|
* valid bit may be set ) have already been zerod by
|
|
|
|
* vm_page_set_validclean().
|
|
|
|
*/
|
|
|
|
for (b = i = 0; i <= PAGE_SIZE / DEV_BSIZE; ++i) {
|
|
|
|
if (i == (PAGE_SIZE / DEV_BSIZE) ||
|
|
|
|
(m->valid & (1 << i))
|
|
|
|
) {
|
|
|
|
if (i > b) {
|
2002-04-15 16:00:03 +00:00
|
|
|
pmap_zero_page_area(m,
|
|
|
|
b << DEV_BSHIFT, (i - b) << DEV_BSHIFT);
|
1999-04-05 19:38:30 +00:00
|
|
|
}
|
|
|
|
b = i + 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* setvalid is TRUE when we can safely set the zero'd areas
|
|
|
|
* as being valid. We can do this if there are no cache consistancy
|
|
|
|
* issues. e.g. it is ok to do with UFS, but not ok to do with NFS.
|
|
|
|
*/
|
|
|
|
if (setvalid)
|
|
|
|
m->valid = VM_PAGE_BITS_ALL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_page_is_valid:
|
|
|
|
*
|
|
|
|
* Is (partial) page valid? Note that the case where size == 0
|
|
|
|
* will return FALSE in the degenerate case where the page is
|
|
|
|
* entirely invalid, and TRUE otherwise.
|
|
|
|
*
|
|
|
|
* May not block.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*/
|
|
|
|
int
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_is_valid(vm_page_t m, int base, int size)
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
{
|
1995-04-09 06:03:56 +00:00
|
|
|
int bits = vm_page_bits(base, size);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
|
1995-04-09 06:03:56 +00:00
|
|
|
if (m->valid && ((m->valid & bits) == bits))
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
return 1;
|
|
|
|
else
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
1998-12-23 01:52:47 +00:00
|
|
|
/*
|
|
|
|
* update dirty bits from pmap/mmu. May not block.
|
|
|
|
*/
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
void
|
2001-07-04 20:15:18 +00:00
|
|
|
vm_page_test_dirty(vm_page_t m)
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
{
|
2000-05-21 12:50:18 +00:00
|
|
|
if ((m->dirty != VM_PAGE_BITS_ALL) && pmap_is_modified(m)) {
|
1999-01-24 06:00:31 +00:00
|
|
|
vm_page_dirty(m);
|
1996-07-30 03:08:57 +00:00
|
|
|
}
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
}
|
|
|
|
|
At long last, commit the zero copy sockets code.
MAKEDEV: Add MAKEDEV glue for the ti(4) device nodes.
ti.4: Update the ti(4) man page to include information on the
TI_JUMBO_HDRSPLIT and TI_PRIVATE_JUMBOS kernel options,
and also include information about the new character
device interface and the associated ioctls.
man9/Makefile: Add jumbo.9 and zero_copy.9 man pages and associated
links.
jumbo.9: New man page describing the jumbo buffer allocator
interface and operation.
zero_copy.9: New man page describing the general characteristics of
the zero copy send and receive code, and what an
application author should do to take advantage of the
zero copy functionality.
NOTES: Add entries for ZERO_COPY_SOCKETS, TI_PRIVATE_JUMBOS,
TI_JUMBO_HDRSPLIT, MSIZE, and MCLSHIFT.
conf/files: Add uipc_jumbo.c and uipc_cow.c.
conf/options: Add the 5 options mentioned above.
kern_subr.c: Receive side zero copy implementation. This takes
"disposable" pages attached to an mbuf, gives them to
a user process, and then recycles the user's page.
This is only active when ZERO_COPY_SOCKETS is turned on
and the kern.ipc.zero_copy.receive sysctl variable is
set to 1.
uipc_cow.c: Send side zero copy functions. Takes a page written
by the user and maps it copy on write and assigns it
kernel virtual address space. Removes copy on write
mapping once the buffer has been freed by the network
stack.
uipc_jumbo.c: Jumbo disposable page allocator code. This allocates
(optionally) disposable pages for network drivers that
want to give the user the option of doing zero copy
receive.
uipc_socket.c: Add kern.ipc.zero_copy.{send,receive} sysctls that are
enabled if ZERO_COPY_SOCKETS is turned on.
Add zero copy send support to sosend() -- pages get
mapped into the kernel instead of getting copied if
they meet size and alignment restrictions.
uipc_syscalls.c:Un-staticize some of the sf* functions so that they
can be used elsewhere. (uipc_cow.c)
if_media.c: In the SIOCGIFMEDIA ioctl in ifmedia_ioctl(), avoid
calling malloc() with M_WAITOK. Return an error if
the M_NOWAIT malloc fails.
The ti(4) driver and the wi(4) driver, at least, call
this with a mutex held. This causes witness warnings
for 'ifconfig -a' with a wi(4) or ti(4) board in the
system. (I've only verified for ti(4)).
ip_output.c: Fragment large datagrams so that each segment contains
a multiple of PAGE_SIZE amount of data plus headers.
This allows the receiver to potentially do page
flipping on receives.
if_ti.c: Add zero copy receive support to the ti(4) driver. If
TI_PRIVATE_JUMBOS is not defined, it now uses the
jumbo(9) buffer allocator for jumbo receive buffers.
Add a new character device interface for the ti(4)
driver for the new debugging interface. This allows
(a patched version of) gdb to talk to the Tigon board
and debug the firmware. There are also a few additional
debugging ioctls available through this interface.
Add header splitting support to the ti(4) driver.
Tweak some of the default interrupt coalescing
parameters to more useful defaults.
Add hooks for supporting transmit flow control, but
leave it turned off with a comment describing why it
is turned off.
if_tireg.h: Change the firmware rev to 12.4.11, since we're really
at 12.4.11 plus fixes from 12.4.13.
Add defines needed for debugging.
Remove the ti_stats structure, it is now defined in
sys/tiio.h.
ti_fw.h: 12.4.11 firmware.
ti_fw2.h: 12.4.11 firmware, plus selected fixes from 12.4.13,
and my header splitting patches. Revision 12.4.13
doesn't handle 10/100 negotiation properly. (This
firmware is the same as what was in the tree previously,
with the addition of header splitting support.)
sys/jumbo.h: Jumbo buffer allocator interface.
sys/mbuf.h: Add a new external mbuf type, EXT_DISPOSABLE, to
indicate that the payload buffer can be thrown away /
flipped to a userland process.
socketvar.h: Add prototype for socow_setup.
tiio.h: ioctl interface to the character portion of the ti(4)
driver, plus associated structure/type definitions.
uio.h: Change prototype for uiomoveco() so that we'll know
whether the source page is disposable.
ufs_readwrite.c:Update for new prototype of uiomoveco().
vm_fault.c: In vm_fault(), check to see whether we need to do a page
based copy on write fault.
vm_object.c: Add a new function, vm_object_allocate_wait(). This
does the same thing that vm_object allocate does, except
that it gives the caller the opportunity to specify whether
it should wait on the uma_zalloc() of the object structre.
This allows vm objects to be allocated while holding a
mutex. (Without generating WITNESS warnings.)
vm_object_allocate() is implemented as a call to
vm_object_allocate_wait() with the malloc flag set to
M_WAITOK.
vm_object.h: Add prototype for vm_object_allocate_wait().
vm_page.c: Add page-based copy on write setup, clear and fault
routines.
vm_page.h: Add page based COW function prototypes and variable in
the vm_page structure.
Many thanks to Drew Gallatin, who wrote the zero copy send and receive
code, and to all the other folks who have tested and reviewed this code
over the years.
2002-06-26 03:37:47 +00:00
|
|
|
int so_zerocp_fullpage = 0;
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_cowfault(vm_page_t m)
|
|
|
|
{
|
|
|
|
vm_page_t mnew;
|
|
|
|
vm_object_t object;
|
|
|
|
vm_pindex_t pindex;
|
|
|
|
|
|
|
|
object = m->object;
|
|
|
|
pindex = m->pindex;
|
|
|
|
vm_page_busy(m);
|
|
|
|
|
|
|
|
retry_alloc:
|
|
|
|
vm_page_remove(m);
|
2002-10-19 18:34:39 +00:00
|
|
|
/*
|
|
|
|
* An interrupt allocation is requested because the page
|
|
|
|
* queues lock is held.
|
|
|
|
*/
|
|
|
|
mnew = vm_page_alloc(object, pindex, VM_ALLOC_INTERRUPT);
|
At long last, commit the zero copy sockets code.
MAKEDEV: Add MAKEDEV glue for the ti(4) device nodes.
ti.4: Update the ti(4) man page to include information on the
TI_JUMBO_HDRSPLIT and TI_PRIVATE_JUMBOS kernel options,
and also include information about the new character
device interface and the associated ioctls.
man9/Makefile: Add jumbo.9 and zero_copy.9 man pages and associated
links.
jumbo.9: New man page describing the jumbo buffer allocator
interface and operation.
zero_copy.9: New man page describing the general characteristics of
the zero copy send and receive code, and what an
application author should do to take advantage of the
zero copy functionality.
NOTES: Add entries for ZERO_COPY_SOCKETS, TI_PRIVATE_JUMBOS,
TI_JUMBO_HDRSPLIT, MSIZE, and MCLSHIFT.
conf/files: Add uipc_jumbo.c and uipc_cow.c.
conf/options: Add the 5 options mentioned above.
kern_subr.c: Receive side zero copy implementation. This takes
"disposable" pages attached to an mbuf, gives them to
a user process, and then recycles the user's page.
This is only active when ZERO_COPY_SOCKETS is turned on
and the kern.ipc.zero_copy.receive sysctl variable is
set to 1.
uipc_cow.c: Send side zero copy functions. Takes a page written
by the user and maps it copy on write and assigns it
kernel virtual address space. Removes copy on write
mapping once the buffer has been freed by the network
stack.
uipc_jumbo.c: Jumbo disposable page allocator code. This allocates
(optionally) disposable pages for network drivers that
want to give the user the option of doing zero copy
receive.
uipc_socket.c: Add kern.ipc.zero_copy.{send,receive} sysctls that are
enabled if ZERO_COPY_SOCKETS is turned on.
Add zero copy send support to sosend() -- pages get
mapped into the kernel instead of getting copied if
they meet size and alignment restrictions.
uipc_syscalls.c:Un-staticize some of the sf* functions so that they
can be used elsewhere. (uipc_cow.c)
if_media.c: In the SIOCGIFMEDIA ioctl in ifmedia_ioctl(), avoid
calling malloc() with M_WAITOK. Return an error if
the M_NOWAIT malloc fails.
The ti(4) driver and the wi(4) driver, at least, call
this with a mutex held. This causes witness warnings
for 'ifconfig -a' with a wi(4) or ti(4) board in the
system. (I've only verified for ti(4)).
ip_output.c: Fragment large datagrams so that each segment contains
a multiple of PAGE_SIZE amount of data plus headers.
This allows the receiver to potentially do page
flipping on receives.
if_ti.c: Add zero copy receive support to the ti(4) driver. If
TI_PRIVATE_JUMBOS is not defined, it now uses the
jumbo(9) buffer allocator for jumbo receive buffers.
Add a new character device interface for the ti(4)
driver for the new debugging interface. This allows
(a patched version of) gdb to talk to the Tigon board
and debug the firmware. There are also a few additional
debugging ioctls available through this interface.
Add header splitting support to the ti(4) driver.
Tweak some of the default interrupt coalescing
parameters to more useful defaults.
Add hooks for supporting transmit flow control, but
leave it turned off with a comment describing why it
is turned off.
if_tireg.h: Change the firmware rev to 12.4.11, since we're really
at 12.4.11 plus fixes from 12.4.13.
Add defines needed for debugging.
Remove the ti_stats structure, it is now defined in
sys/tiio.h.
ti_fw.h: 12.4.11 firmware.
ti_fw2.h: 12.4.11 firmware, plus selected fixes from 12.4.13,
and my header splitting patches. Revision 12.4.13
doesn't handle 10/100 negotiation properly. (This
firmware is the same as what was in the tree previously,
with the addition of header splitting support.)
sys/jumbo.h: Jumbo buffer allocator interface.
sys/mbuf.h: Add a new external mbuf type, EXT_DISPOSABLE, to
indicate that the payload buffer can be thrown away /
flipped to a userland process.
socketvar.h: Add prototype for socow_setup.
tiio.h: ioctl interface to the character portion of the ti(4)
driver, plus associated structure/type definitions.
uio.h: Change prototype for uiomoveco() so that we'll know
whether the source page is disposable.
ufs_readwrite.c:Update for new prototype of uiomoveco().
vm_fault.c: In vm_fault(), check to see whether we need to do a page
based copy on write fault.
vm_object.c: Add a new function, vm_object_allocate_wait(). This
does the same thing that vm_object allocate does, except
that it gives the caller the opportunity to specify whether
it should wait on the uma_zalloc() of the object structre.
This allows vm objects to be allocated while holding a
mutex. (Without generating WITNESS warnings.)
vm_object_allocate() is implemented as a call to
vm_object_allocate_wait() with the malloc flag set to
M_WAITOK.
vm_object.h: Add prototype for vm_object_allocate_wait().
vm_page.c: Add page-based copy on write setup, clear and fault
routines.
vm_page.h: Add page based COW function prototypes and variable in
the vm_page structure.
Many thanks to Drew Gallatin, who wrote the zero copy send and receive
code, and to all the other folks who have tested and reviewed this code
over the years.
2002-06-26 03:37:47 +00:00
|
|
|
if (mnew == NULL) {
|
|
|
|
vm_page_insert(m, object, pindex);
|
2002-10-19 18:34:39 +00:00
|
|
|
vm_page_unlock_queues();
|
2003-06-22 21:35:41 +00:00
|
|
|
VM_OBJECT_UNLOCK(object);
|
At long last, commit the zero copy sockets code.
MAKEDEV: Add MAKEDEV glue for the ti(4) device nodes.
ti.4: Update the ti(4) man page to include information on the
TI_JUMBO_HDRSPLIT and TI_PRIVATE_JUMBOS kernel options,
and also include information about the new character
device interface and the associated ioctls.
man9/Makefile: Add jumbo.9 and zero_copy.9 man pages and associated
links.
jumbo.9: New man page describing the jumbo buffer allocator
interface and operation.
zero_copy.9: New man page describing the general characteristics of
the zero copy send and receive code, and what an
application author should do to take advantage of the
zero copy functionality.
NOTES: Add entries for ZERO_COPY_SOCKETS, TI_PRIVATE_JUMBOS,
TI_JUMBO_HDRSPLIT, MSIZE, and MCLSHIFT.
conf/files: Add uipc_jumbo.c and uipc_cow.c.
conf/options: Add the 5 options mentioned above.
kern_subr.c: Receive side zero copy implementation. This takes
"disposable" pages attached to an mbuf, gives them to
a user process, and then recycles the user's page.
This is only active when ZERO_COPY_SOCKETS is turned on
and the kern.ipc.zero_copy.receive sysctl variable is
set to 1.
uipc_cow.c: Send side zero copy functions. Takes a page written
by the user and maps it copy on write and assigns it
kernel virtual address space. Removes copy on write
mapping once the buffer has been freed by the network
stack.
uipc_jumbo.c: Jumbo disposable page allocator code. This allocates
(optionally) disposable pages for network drivers that
want to give the user the option of doing zero copy
receive.
uipc_socket.c: Add kern.ipc.zero_copy.{send,receive} sysctls that are
enabled if ZERO_COPY_SOCKETS is turned on.
Add zero copy send support to sosend() -- pages get
mapped into the kernel instead of getting copied if
they meet size and alignment restrictions.
uipc_syscalls.c:Un-staticize some of the sf* functions so that they
can be used elsewhere. (uipc_cow.c)
if_media.c: In the SIOCGIFMEDIA ioctl in ifmedia_ioctl(), avoid
calling malloc() with M_WAITOK. Return an error if
the M_NOWAIT malloc fails.
The ti(4) driver and the wi(4) driver, at least, call
this with a mutex held. This causes witness warnings
for 'ifconfig -a' with a wi(4) or ti(4) board in the
system. (I've only verified for ti(4)).
ip_output.c: Fragment large datagrams so that each segment contains
a multiple of PAGE_SIZE amount of data plus headers.
This allows the receiver to potentially do page
flipping on receives.
if_ti.c: Add zero copy receive support to the ti(4) driver. If
TI_PRIVATE_JUMBOS is not defined, it now uses the
jumbo(9) buffer allocator for jumbo receive buffers.
Add a new character device interface for the ti(4)
driver for the new debugging interface. This allows
(a patched version of) gdb to talk to the Tigon board
and debug the firmware. There are also a few additional
debugging ioctls available through this interface.
Add header splitting support to the ti(4) driver.
Tweak some of the default interrupt coalescing
parameters to more useful defaults.
Add hooks for supporting transmit flow control, but
leave it turned off with a comment describing why it
is turned off.
if_tireg.h: Change the firmware rev to 12.4.11, since we're really
at 12.4.11 plus fixes from 12.4.13.
Add defines needed for debugging.
Remove the ti_stats structure, it is now defined in
sys/tiio.h.
ti_fw.h: 12.4.11 firmware.
ti_fw2.h: 12.4.11 firmware, plus selected fixes from 12.4.13,
and my header splitting patches. Revision 12.4.13
doesn't handle 10/100 negotiation properly. (This
firmware is the same as what was in the tree previously,
with the addition of header splitting support.)
sys/jumbo.h: Jumbo buffer allocator interface.
sys/mbuf.h: Add a new external mbuf type, EXT_DISPOSABLE, to
indicate that the payload buffer can be thrown away /
flipped to a userland process.
socketvar.h: Add prototype for socow_setup.
tiio.h: ioctl interface to the character portion of the ti(4)
driver, plus associated structure/type definitions.
uio.h: Change prototype for uiomoveco() so that we'll know
whether the source page is disposable.
ufs_readwrite.c:Update for new prototype of uiomoveco().
vm_fault.c: In vm_fault(), check to see whether we need to do a page
based copy on write fault.
vm_object.c: Add a new function, vm_object_allocate_wait(). This
does the same thing that vm_object allocate does, except
that it gives the caller the opportunity to specify whether
it should wait on the uma_zalloc() of the object structre.
This allows vm objects to be allocated while holding a
mutex. (Without generating WITNESS warnings.)
vm_object_allocate() is implemented as a call to
vm_object_allocate_wait() with the malloc flag set to
M_WAITOK.
vm_object.h: Add prototype for vm_object_allocate_wait().
vm_page.c: Add page-based copy on write setup, clear and fault
routines.
vm_page.h: Add page based COW function prototypes and variable in
the vm_page structure.
Many thanks to Drew Gallatin, who wrote the zero copy send and receive
code, and to all the other folks who have tested and reviewed this code
over the years.
2002-06-26 03:37:47 +00:00
|
|
|
VM_WAIT;
|
2003-06-22 21:35:41 +00:00
|
|
|
VM_OBJECT_LOCK(object);
|
2002-10-19 18:34:39 +00:00
|
|
|
vm_page_lock_queues();
|
At long last, commit the zero copy sockets code.
MAKEDEV: Add MAKEDEV glue for the ti(4) device nodes.
ti.4: Update the ti(4) man page to include information on the
TI_JUMBO_HDRSPLIT and TI_PRIVATE_JUMBOS kernel options,
and also include information about the new character
device interface and the associated ioctls.
man9/Makefile: Add jumbo.9 and zero_copy.9 man pages and associated
links.
jumbo.9: New man page describing the jumbo buffer allocator
interface and operation.
zero_copy.9: New man page describing the general characteristics of
the zero copy send and receive code, and what an
application author should do to take advantage of the
zero copy functionality.
NOTES: Add entries for ZERO_COPY_SOCKETS, TI_PRIVATE_JUMBOS,
TI_JUMBO_HDRSPLIT, MSIZE, and MCLSHIFT.
conf/files: Add uipc_jumbo.c and uipc_cow.c.
conf/options: Add the 5 options mentioned above.
kern_subr.c: Receive side zero copy implementation. This takes
"disposable" pages attached to an mbuf, gives them to
a user process, and then recycles the user's page.
This is only active when ZERO_COPY_SOCKETS is turned on
and the kern.ipc.zero_copy.receive sysctl variable is
set to 1.
uipc_cow.c: Send side zero copy functions. Takes a page written
by the user and maps it copy on write and assigns it
kernel virtual address space. Removes copy on write
mapping once the buffer has been freed by the network
stack.
uipc_jumbo.c: Jumbo disposable page allocator code. This allocates
(optionally) disposable pages for network drivers that
want to give the user the option of doing zero copy
receive.
uipc_socket.c: Add kern.ipc.zero_copy.{send,receive} sysctls that are
enabled if ZERO_COPY_SOCKETS is turned on.
Add zero copy send support to sosend() -- pages get
mapped into the kernel instead of getting copied if
they meet size and alignment restrictions.
uipc_syscalls.c:Un-staticize some of the sf* functions so that they
can be used elsewhere. (uipc_cow.c)
if_media.c: In the SIOCGIFMEDIA ioctl in ifmedia_ioctl(), avoid
calling malloc() with M_WAITOK. Return an error if
the M_NOWAIT malloc fails.
The ti(4) driver and the wi(4) driver, at least, call
this with a mutex held. This causes witness warnings
for 'ifconfig -a' with a wi(4) or ti(4) board in the
system. (I've only verified for ti(4)).
ip_output.c: Fragment large datagrams so that each segment contains
a multiple of PAGE_SIZE amount of data plus headers.
This allows the receiver to potentially do page
flipping on receives.
if_ti.c: Add zero copy receive support to the ti(4) driver. If
TI_PRIVATE_JUMBOS is not defined, it now uses the
jumbo(9) buffer allocator for jumbo receive buffers.
Add a new character device interface for the ti(4)
driver for the new debugging interface. This allows
(a patched version of) gdb to talk to the Tigon board
and debug the firmware. There are also a few additional
debugging ioctls available through this interface.
Add header splitting support to the ti(4) driver.
Tweak some of the default interrupt coalescing
parameters to more useful defaults.
Add hooks for supporting transmit flow control, but
leave it turned off with a comment describing why it
is turned off.
if_tireg.h: Change the firmware rev to 12.4.11, since we're really
at 12.4.11 plus fixes from 12.4.13.
Add defines needed for debugging.
Remove the ti_stats structure, it is now defined in
sys/tiio.h.
ti_fw.h: 12.4.11 firmware.
ti_fw2.h: 12.4.11 firmware, plus selected fixes from 12.4.13,
and my header splitting patches. Revision 12.4.13
doesn't handle 10/100 negotiation properly. (This
firmware is the same as what was in the tree previously,
with the addition of header splitting support.)
sys/jumbo.h: Jumbo buffer allocator interface.
sys/mbuf.h: Add a new external mbuf type, EXT_DISPOSABLE, to
indicate that the payload buffer can be thrown away /
flipped to a userland process.
socketvar.h: Add prototype for socow_setup.
tiio.h: ioctl interface to the character portion of the ti(4)
driver, plus associated structure/type definitions.
uio.h: Change prototype for uiomoveco() so that we'll know
whether the source page is disposable.
ufs_readwrite.c:Update for new prototype of uiomoveco().
vm_fault.c: In vm_fault(), check to see whether we need to do a page
based copy on write fault.
vm_object.c: Add a new function, vm_object_allocate_wait(). This
does the same thing that vm_object allocate does, except
that it gives the caller the opportunity to specify whether
it should wait on the uma_zalloc() of the object structre.
This allows vm objects to be allocated while holding a
mutex. (Without generating WITNESS warnings.)
vm_object_allocate() is implemented as a call to
vm_object_allocate_wait() with the malloc flag set to
M_WAITOK.
vm_object.h: Add prototype for vm_object_allocate_wait().
vm_page.c: Add page-based copy on write setup, clear and fault
routines.
vm_page.h: Add page based COW function prototypes and variable in
the vm_page structure.
Many thanks to Drew Gallatin, who wrote the zero copy send and receive
code, and to all the other folks who have tested and reviewed this code
over the years.
2002-06-26 03:37:47 +00:00
|
|
|
goto retry_alloc;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (m->cow == 0) {
|
|
|
|
/*
|
|
|
|
* check to see if we raced with an xmit complete when
|
|
|
|
* waiting to allocate a page. If so, put things back
|
|
|
|
* the way they were
|
|
|
|
*/
|
|
|
|
vm_page_busy(mnew);
|
|
|
|
vm_page_free(mnew);
|
|
|
|
vm_page_insert(m, object, pindex);
|
|
|
|
} else { /* clear COW & copy page */
|
|
|
|
if (so_zerocp_fullpage) {
|
|
|
|
mnew->valid = VM_PAGE_BITS_ALL;
|
|
|
|
} else {
|
|
|
|
vm_page_copy(m, mnew);
|
|
|
|
}
|
|
|
|
vm_page_dirty(mnew);
|
|
|
|
vm_page_flag_clear(mnew, PG_BUSY);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_cowclear(vm_page_t m)
|
|
|
|
{
|
|
|
|
|
2002-09-02 04:04:12 +00:00
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
At long last, commit the zero copy sockets code.
MAKEDEV: Add MAKEDEV glue for the ti(4) device nodes.
ti.4: Update the ti(4) man page to include information on the
TI_JUMBO_HDRSPLIT and TI_PRIVATE_JUMBOS kernel options,
and also include information about the new character
device interface and the associated ioctls.
man9/Makefile: Add jumbo.9 and zero_copy.9 man pages and associated
links.
jumbo.9: New man page describing the jumbo buffer allocator
interface and operation.
zero_copy.9: New man page describing the general characteristics of
the zero copy send and receive code, and what an
application author should do to take advantage of the
zero copy functionality.
NOTES: Add entries for ZERO_COPY_SOCKETS, TI_PRIVATE_JUMBOS,
TI_JUMBO_HDRSPLIT, MSIZE, and MCLSHIFT.
conf/files: Add uipc_jumbo.c and uipc_cow.c.
conf/options: Add the 5 options mentioned above.
kern_subr.c: Receive side zero copy implementation. This takes
"disposable" pages attached to an mbuf, gives them to
a user process, and then recycles the user's page.
This is only active when ZERO_COPY_SOCKETS is turned on
and the kern.ipc.zero_copy.receive sysctl variable is
set to 1.
uipc_cow.c: Send side zero copy functions. Takes a page written
by the user and maps it copy on write and assigns it
kernel virtual address space. Removes copy on write
mapping once the buffer has been freed by the network
stack.
uipc_jumbo.c: Jumbo disposable page allocator code. This allocates
(optionally) disposable pages for network drivers that
want to give the user the option of doing zero copy
receive.
uipc_socket.c: Add kern.ipc.zero_copy.{send,receive} sysctls that are
enabled if ZERO_COPY_SOCKETS is turned on.
Add zero copy send support to sosend() -- pages get
mapped into the kernel instead of getting copied if
they meet size and alignment restrictions.
uipc_syscalls.c:Un-staticize some of the sf* functions so that they
can be used elsewhere. (uipc_cow.c)
if_media.c: In the SIOCGIFMEDIA ioctl in ifmedia_ioctl(), avoid
calling malloc() with M_WAITOK. Return an error if
the M_NOWAIT malloc fails.
The ti(4) driver and the wi(4) driver, at least, call
this with a mutex held. This causes witness warnings
for 'ifconfig -a' with a wi(4) or ti(4) board in the
system. (I've only verified for ti(4)).
ip_output.c: Fragment large datagrams so that each segment contains
a multiple of PAGE_SIZE amount of data plus headers.
This allows the receiver to potentially do page
flipping on receives.
if_ti.c: Add zero copy receive support to the ti(4) driver. If
TI_PRIVATE_JUMBOS is not defined, it now uses the
jumbo(9) buffer allocator for jumbo receive buffers.
Add a new character device interface for the ti(4)
driver for the new debugging interface. This allows
(a patched version of) gdb to talk to the Tigon board
and debug the firmware. There are also a few additional
debugging ioctls available through this interface.
Add header splitting support to the ti(4) driver.
Tweak some of the default interrupt coalescing
parameters to more useful defaults.
Add hooks for supporting transmit flow control, but
leave it turned off with a comment describing why it
is turned off.
if_tireg.h: Change the firmware rev to 12.4.11, since we're really
at 12.4.11 plus fixes from 12.4.13.
Add defines needed for debugging.
Remove the ti_stats structure, it is now defined in
sys/tiio.h.
ti_fw.h: 12.4.11 firmware.
ti_fw2.h: 12.4.11 firmware, plus selected fixes from 12.4.13,
and my header splitting patches. Revision 12.4.13
doesn't handle 10/100 negotiation properly. (This
firmware is the same as what was in the tree previously,
with the addition of header splitting support.)
sys/jumbo.h: Jumbo buffer allocator interface.
sys/mbuf.h: Add a new external mbuf type, EXT_DISPOSABLE, to
indicate that the payload buffer can be thrown away /
flipped to a userland process.
socketvar.h: Add prototype for socow_setup.
tiio.h: ioctl interface to the character portion of the ti(4)
driver, plus associated structure/type definitions.
uio.h: Change prototype for uiomoveco() so that we'll know
whether the source page is disposable.
ufs_readwrite.c:Update for new prototype of uiomoveco().
vm_fault.c: In vm_fault(), check to see whether we need to do a page
based copy on write fault.
vm_object.c: Add a new function, vm_object_allocate_wait(). This
does the same thing that vm_object allocate does, except
that it gives the caller the opportunity to specify whether
it should wait on the uma_zalloc() of the object structre.
This allows vm objects to be allocated while holding a
mutex. (Without generating WITNESS warnings.)
vm_object_allocate() is implemented as a call to
vm_object_allocate_wait() with the malloc flag set to
M_WAITOK.
vm_object.h: Add prototype for vm_object_allocate_wait().
vm_page.c: Add page-based copy on write setup, clear and fault
routines.
vm_page.h: Add page based COW function prototypes and variable in
the vm_page structure.
Many thanks to Drew Gallatin, who wrote the zero copy send and receive
code, and to all the other folks who have tested and reviewed this code
over the years.
2002-06-26 03:37:47 +00:00
|
|
|
if (m->cow) {
|
2002-09-02 04:04:12 +00:00
|
|
|
m->cow--;
|
At long last, commit the zero copy sockets code.
MAKEDEV: Add MAKEDEV glue for the ti(4) device nodes.
ti.4: Update the ti(4) man page to include information on the
TI_JUMBO_HDRSPLIT and TI_PRIVATE_JUMBOS kernel options,
and also include information about the new character
device interface and the associated ioctls.
man9/Makefile: Add jumbo.9 and zero_copy.9 man pages and associated
links.
jumbo.9: New man page describing the jumbo buffer allocator
interface and operation.
zero_copy.9: New man page describing the general characteristics of
the zero copy send and receive code, and what an
application author should do to take advantage of the
zero copy functionality.
NOTES: Add entries for ZERO_COPY_SOCKETS, TI_PRIVATE_JUMBOS,
TI_JUMBO_HDRSPLIT, MSIZE, and MCLSHIFT.
conf/files: Add uipc_jumbo.c and uipc_cow.c.
conf/options: Add the 5 options mentioned above.
kern_subr.c: Receive side zero copy implementation. This takes
"disposable" pages attached to an mbuf, gives them to
a user process, and then recycles the user's page.
This is only active when ZERO_COPY_SOCKETS is turned on
and the kern.ipc.zero_copy.receive sysctl variable is
set to 1.
uipc_cow.c: Send side zero copy functions. Takes a page written
by the user and maps it copy on write and assigns it
kernel virtual address space. Removes copy on write
mapping once the buffer has been freed by the network
stack.
uipc_jumbo.c: Jumbo disposable page allocator code. This allocates
(optionally) disposable pages for network drivers that
want to give the user the option of doing zero copy
receive.
uipc_socket.c: Add kern.ipc.zero_copy.{send,receive} sysctls that are
enabled if ZERO_COPY_SOCKETS is turned on.
Add zero copy send support to sosend() -- pages get
mapped into the kernel instead of getting copied if
they meet size and alignment restrictions.
uipc_syscalls.c:Un-staticize some of the sf* functions so that they
can be used elsewhere. (uipc_cow.c)
if_media.c: In the SIOCGIFMEDIA ioctl in ifmedia_ioctl(), avoid
calling malloc() with M_WAITOK. Return an error if
the M_NOWAIT malloc fails.
The ti(4) driver and the wi(4) driver, at least, call
this with a mutex held. This causes witness warnings
for 'ifconfig -a' with a wi(4) or ti(4) board in the
system. (I've only verified for ti(4)).
ip_output.c: Fragment large datagrams so that each segment contains
a multiple of PAGE_SIZE amount of data plus headers.
This allows the receiver to potentially do page
flipping on receives.
if_ti.c: Add zero copy receive support to the ti(4) driver. If
TI_PRIVATE_JUMBOS is not defined, it now uses the
jumbo(9) buffer allocator for jumbo receive buffers.
Add a new character device interface for the ti(4)
driver for the new debugging interface. This allows
(a patched version of) gdb to talk to the Tigon board
and debug the firmware. There are also a few additional
debugging ioctls available through this interface.
Add header splitting support to the ti(4) driver.
Tweak some of the default interrupt coalescing
parameters to more useful defaults.
Add hooks for supporting transmit flow control, but
leave it turned off with a comment describing why it
is turned off.
if_tireg.h: Change the firmware rev to 12.4.11, since we're really
at 12.4.11 plus fixes from 12.4.13.
Add defines needed for debugging.
Remove the ti_stats structure, it is now defined in
sys/tiio.h.
ti_fw.h: 12.4.11 firmware.
ti_fw2.h: 12.4.11 firmware, plus selected fixes from 12.4.13,
and my header splitting patches. Revision 12.4.13
doesn't handle 10/100 negotiation properly. (This
firmware is the same as what was in the tree previously,
with the addition of header splitting support.)
sys/jumbo.h: Jumbo buffer allocator interface.
sys/mbuf.h: Add a new external mbuf type, EXT_DISPOSABLE, to
indicate that the payload buffer can be thrown away /
flipped to a userland process.
socketvar.h: Add prototype for socow_setup.
tiio.h: ioctl interface to the character portion of the ti(4)
driver, plus associated structure/type definitions.
uio.h: Change prototype for uiomoveco() so that we'll know
whether the source page is disposable.
ufs_readwrite.c:Update for new prototype of uiomoveco().
vm_fault.c: In vm_fault(), check to see whether we need to do a page
based copy on write fault.
vm_object.c: Add a new function, vm_object_allocate_wait(). This
does the same thing that vm_object allocate does, except
that it gives the caller the opportunity to specify whether
it should wait on the uma_zalloc() of the object structre.
This allows vm objects to be allocated while holding a
mutex. (Without generating WITNESS warnings.)
vm_object_allocate() is implemented as a call to
vm_object_allocate_wait() with the malloc flag set to
M_WAITOK.
vm_object.h: Add prototype for vm_object_allocate_wait().
vm_page.c: Add page-based copy on write setup, clear and fault
routines.
vm_page.h: Add page based COW function prototypes and variable in
the vm_page structure.
Many thanks to Drew Gallatin, who wrote the zero copy send and receive
code, and to all the other folks who have tested and reviewed this code
over the years.
2002-06-26 03:37:47 +00:00
|
|
|
/*
|
|
|
|
* let vm_fault add back write permission lazily
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* sf_buf_free() will free the page, so we needn't do it here
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
|
|
|
|
void
|
|
|
|
vm_page_cowsetup(vm_page_t m)
|
|
|
|
{
|
2002-09-02 04:04:12 +00:00
|
|
|
|
|
|
|
mtx_assert(&vm_page_queue_mtx, MA_OWNED);
|
|
|
|
m->cow++;
|
2002-11-18 04:05:22 +00:00
|
|
|
pmap_page_protect(m, VM_PROT_READ);
|
At long last, commit the zero copy sockets code.
MAKEDEV: Add MAKEDEV glue for the ti(4) device nodes.
ti.4: Update the ti(4) man page to include information on the
TI_JUMBO_HDRSPLIT and TI_PRIVATE_JUMBOS kernel options,
and also include information about the new character
device interface and the associated ioctls.
man9/Makefile: Add jumbo.9 and zero_copy.9 man pages and associated
links.
jumbo.9: New man page describing the jumbo buffer allocator
interface and operation.
zero_copy.9: New man page describing the general characteristics of
the zero copy send and receive code, and what an
application author should do to take advantage of the
zero copy functionality.
NOTES: Add entries for ZERO_COPY_SOCKETS, TI_PRIVATE_JUMBOS,
TI_JUMBO_HDRSPLIT, MSIZE, and MCLSHIFT.
conf/files: Add uipc_jumbo.c and uipc_cow.c.
conf/options: Add the 5 options mentioned above.
kern_subr.c: Receive side zero copy implementation. This takes
"disposable" pages attached to an mbuf, gives them to
a user process, and then recycles the user's page.
This is only active when ZERO_COPY_SOCKETS is turned on
and the kern.ipc.zero_copy.receive sysctl variable is
set to 1.
uipc_cow.c: Send side zero copy functions. Takes a page written
by the user and maps it copy on write and assigns it
kernel virtual address space. Removes copy on write
mapping once the buffer has been freed by the network
stack.
uipc_jumbo.c: Jumbo disposable page allocator code. This allocates
(optionally) disposable pages for network drivers that
want to give the user the option of doing zero copy
receive.
uipc_socket.c: Add kern.ipc.zero_copy.{send,receive} sysctls that are
enabled if ZERO_COPY_SOCKETS is turned on.
Add zero copy send support to sosend() -- pages get
mapped into the kernel instead of getting copied if
they meet size and alignment restrictions.
uipc_syscalls.c:Un-staticize some of the sf* functions so that they
can be used elsewhere. (uipc_cow.c)
if_media.c: In the SIOCGIFMEDIA ioctl in ifmedia_ioctl(), avoid
calling malloc() with M_WAITOK. Return an error if
the M_NOWAIT malloc fails.
The ti(4) driver and the wi(4) driver, at least, call
this with a mutex held. This causes witness warnings
for 'ifconfig -a' with a wi(4) or ti(4) board in the
system. (I've only verified for ti(4)).
ip_output.c: Fragment large datagrams so that each segment contains
a multiple of PAGE_SIZE amount of data plus headers.
This allows the receiver to potentially do page
flipping on receives.
if_ti.c: Add zero copy receive support to the ti(4) driver. If
TI_PRIVATE_JUMBOS is not defined, it now uses the
jumbo(9) buffer allocator for jumbo receive buffers.
Add a new character device interface for the ti(4)
driver for the new debugging interface. This allows
(a patched version of) gdb to talk to the Tigon board
and debug the firmware. There are also a few additional
debugging ioctls available through this interface.
Add header splitting support to the ti(4) driver.
Tweak some of the default interrupt coalescing
parameters to more useful defaults.
Add hooks for supporting transmit flow control, but
leave it turned off with a comment describing why it
is turned off.
if_tireg.h: Change the firmware rev to 12.4.11, since we're really
at 12.4.11 plus fixes from 12.4.13.
Add defines needed for debugging.
Remove the ti_stats structure, it is now defined in
sys/tiio.h.
ti_fw.h: 12.4.11 firmware.
ti_fw2.h: 12.4.11 firmware, plus selected fixes from 12.4.13,
and my header splitting patches. Revision 12.4.13
doesn't handle 10/100 negotiation properly. (This
firmware is the same as what was in the tree previously,
with the addition of header splitting support.)
sys/jumbo.h: Jumbo buffer allocator interface.
sys/mbuf.h: Add a new external mbuf type, EXT_DISPOSABLE, to
indicate that the payload buffer can be thrown away /
flipped to a userland process.
socketvar.h: Add prototype for socow_setup.
tiio.h: ioctl interface to the character portion of the ti(4)
driver, plus associated structure/type definitions.
uio.h: Change prototype for uiomoveco() so that we'll know
whether the source page is disposable.
ufs_readwrite.c:Update for new prototype of uiomoveco().
vm_fault.c: In vm_fault(), check to see whether we need to do a page
based copy on write fault.
vm_object.c: Add a new function, vm_object_allocate_wait(). This
does the same thing that vm_object allocate does, except
that it gives the caller the opportunity to specify whether
it should wait on the uma_zalloc() of the object structre.
This allows vm objects to be allocated while holding a
mutex. (Without generating WITNESS warnings.)
vm_object_allocate() is implemented as a call to
vm_object_allocate_wait() with the malloc flag set to
M_WAITOK.
vm_object.h: Add prototype for vm_object_allocate_wait().
vm_page.c: Add page-based copy on write setup, clear and fault
routines.
vm_page.h: Add page based COW function prototypes and variable in
the vm_page structure.
Many thanks to Drew Gallatin, who wrote the zero copy send and receive
code, and to all the other folks who have tested and reviewed this code
over the years.
2002-06-26 03:37:47 +00:00
|
|
|
}
|
|
|
|
|
1996-09-14 11:54:59 +00:00
|
|
|
#include "opt_ddb.h"
|
1995-04-16 09:59:16 +00:00
|
|
|
#ifdef DDB
|
1996-09-14 11:54:59 +00:00
|
|
|
#include <sys/kernel.h>
|
|
|
|
|
|
|
|
#include <ddb/ddb.h>
|
|
|
|
|
|
|
|
DB_SHOW_COMMAND(page, vm_page_print_page_info)
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
{
|
1996-09-14 11:54:59 +00:00
|
|
|
db_printf("cnt.v_free_count: %d\n", cnt.v_free_count);
|
|
|
|
db_printf("cnt.v_cache_count: %d\n", cnt.v_cache_count);
|
|
|
|
db_printf("cnt.v_inactive_count: %d\n", cnt.v_inactive_count);
|
|
|
|
db_printf("cnt.v_active_count: %d\n", cnt.v_active_count);
|
|
|
|
db_printf("cnt.v_wire_count: %d\n", cnt.v_wire_count);
|
|
|
|
db_printf("cnt.v_free_reserved: %d\n", cnt.v_free_reserved);
|
|
|
|
db_printf("cnt.v_free_min: %d\n", cnt.v_free_min);
|
|
|
|
db_printf("cnt.v_free_target: %d\n", cnt.v_free_target);
|
|
|
|
db_printf("cnt.v_cache_min: %d\n", cnt.v_cache_min);
|
|
|
|
db_printf("cnt.v_inactive_target: %d\n", cnt.v_inactive_target);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
1996-09-08 20:44:49 +00:00
|
|
|
|
1996-09-14 11:54:59 +00:00
|
|
|
DB_SHOW_COMMAND(pageq, vm_page_print_pageq_info)
|
1996-09-08 20:44:49 +00:00
|
|
|
{
|
|
|
|
int i;
|
1996-09-14 11:54:59 +00:00
|
|
|
db_printf("PQ_FREE:");
|
2001-07-04 19:00:13 +00:00
|
|
|
for (i = 0; i < PQ_L2_SIZE; i++) {
|
1999-07-31 18:31:00 +00:00
|
|
|
db_printf(" %d", vm_page_queues[PQ_FREE + i].lcnt);
|
1996-09-08 20:44:49 +00:00
|
|
|
}
|
1996-09-14 11:54:59 +00:00
|
|
|
db_printf("\n");
|
1996-09-08 20:44:49 +00:00
|
|
|
|
1996-09-14 11:54:59 +00:00
|
|
|
db_printf("PQ_CACHE:");
|
2001-07-04 19:00:13 +00:00
|
|
|
for (i = 0; i < PQ_L2_SIZE; i++) {
|
1999-07-31 18:31:00 +00:00
|
|
|
db_printf(" %d", vm_page_queues[PQ_CACHE + i].lcnt);
|
1996-09-08 20:44:49 +00:00
|
|
|
}
|
1996-09-14 11:54:59 +00:00
|
|
|
db_printf("\n");
|
1996-09-08 20:44:49 +00:00
|
|
|
|
1996-09-14 11:54:59 +00:00
|
|
|
db_printf("PQ_ACTIVE: %d, PQ_INACTIVE: %d\n",
|
1999-07-31 18:31:00 +00:00
|
|
|
vm_page_queues[PQ_ACTIVE].lcnt,
|
|
|
|
vm_page_queues[PQ_INACTIVE].lcnt);
|
1996-09-08 20:44:49 +00:00
|
|
|
}
|
1996-09-14 11:54:59 +00:00
|
|
|
#endif /* DDB */
|