2005-01-07 02:29:27 +00:00
|
|
|
/*-
|
2017-11-30 15:48:35 +00:00
|
|
|
* SPDX-License-Identifier: (BSD-3-Clause AND MIT-CMU)
|
2017-11-20 19:43:44 +00:00
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Copyright (c) 1991, 1993
|
|
|
|
* The Regents of the University of California. All rights reserved.
|
|
|
|
*
|
|
|
|
* This code is derived from software contributed to Berkeley by
|
|
|
|
* The Mach Operating System project at Carnegie-Mellon University.
|
|
|
|
*
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
* are met:
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
2017-02-28 23:42:47 +00:00
|
|
|
* 3. Neither the name of the University nor the names of its contributors
|
1994-05-24 10:09:53 +00:00
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
* without specific prior written permission.
|
|
|
|
*
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
*
|
1994-08-02 07:55:43 +00:00
|
|
|
* from: @(#)vm_glue.c 8.6 (Berkeley) 1/5/94
|
1994-05-24 10:09:53 +00:00
|
|
|
*
|
|
|
|
*
|
|
|
|
* Copyright (c) 1987, 1990 Carnegie-Mellon University.
|
|
|
|
* All rights reserved.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Permission to use, copy, modify and distribute this software and
|
|
|
|
* its documentation is hereby granted, provided that both the copyright
|
|
|
|
* notice and this permission notice appear in all copies of the
|
|
|
|
* software, derivative works or modified versions, and any portions
|
|
|
|
* thereof, and that both notices appear in supporting documentation.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
|
|
|
* CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS"
|
|
|
|
* CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND
|
1994-05-24 10:09:53 +00:00
|
|
|
* FOR ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE.
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Carnegie Mellon requests users of this software to return to
|
|
|
|
*
|
|
|
|
* Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU
|
|
|
|
* School of Computer Science
|
|
|
|
* Carnegie Mellon University
|
|
|
|
* Pittsburgh PA 15213-3890
|
|
|
|
*
|
|
|
|
* any improvements or extensions that they make and grant Carnegie the
|
|
|
|
* rights to redistribute these changes.
|
|
|
|
*/
|
|
|
|
|
2003-06-11 23:50:51 +00:00
|
|
|
#include <sys/cdefs.h>
|
1998-09-29 17:33:59 +00:00
|
|
|
#include "opt_vm.h"
|
2003-07-31 01:25:05 +00:00
|
|
|
#include "opt_kstack_pages.h"
|
|
|
|
#include "opt_kstack_max_pages.h"
|
2014-10-04 18:38:14 +00:00
|
|
|
#include "opt_kstack_usage_prof.h"
|
1996-12-22 23:17:09 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/param.h>
|
|
|
|
#include <sys/systm.h>
|
2021-04-13 21:39:55 +00:00
|
|
|
#include <sys/asan.h>
|
2018-10-22 17:04:04 +00:00
|
|
|
#include <sys/domainset.h>
|
2003-04-29 13:36:06 +00:00
|
|
|
#include <sys/limits.h>
|
2001-05-01 08:13:21 +00:00
|
|
|
#include <sys/lock.h>
|
2013-08-07 06:21:20 +00:00
|
|
|
#include <sys/malloc.h>
|
2021-08-10 21:15:03 +00:00
|
|
|
#include <sys/msan.h>
|
2001-05-01 08:13:21 +00:00
|
|
|
#include <sys/mutex.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/proc.h>
|
2011-04-05 20:23:59 +00:00
|
|
|
#include <sys/racct.h>
|
2020-11-04 16:30:56 +00:00
|
|
|
#include <sys/refcount.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/resourcevar.h>
|
2013-03-09 02:32:23 +00:00
|
|
|
#include <sys/rwlock.h>
|
2005-12-16 18:34:14 +00:00
|
|
|
#include <sys/sched.h>
|
|
|
|
#include <sys/sf_buf.h>
|
1995-03-28 07:58:53 +00:00
|
|
|
#include <sys/shm.h>
|
2019-11-22 16:30:47 +00:00
|
|
|
#include <sys/smp.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <sys/vmmeter.h>
|
2013-08-07 06:21:20 +00:00
|
|
|
#include <sys/vmem.h>
|
2001-03-28 11:52:56 +00:00
|
|
|
#include <sys/sx.h>
|
1997-12-06 02:23:36 +00:00
|
|
|
#include <sys/sysctl.h>
|
1994-05-25 09:21:21 +00:00
|
|
|
#include <sys/kernel.h>
|
2000-09-07 01:33:02 +00:00
|
|
|
#include <sys/ktr.h>
|
1997-04-07 07:16:06 +00:00
|
|
|
#include <sys/unistd.h>
|
1994-05-25 09:21:21 +00:00
|
|
|
|
2019-08-06 23:15:34 +00:00
|
|
|
#include <vm/uma.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_param.h>
|
|
|
|
#include <vm/pmap.h>
|
2018-09-06 19:28:52 +00:00
|
|
|
#include <vm/vm_domainset.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_map.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm_page.h>
|
1994-05-25 09:21:21 +00:00
|
|
|
#include <vm/vm_pageout.h>
|
2002-07-07 23:05:27 +00:00
|
|
|
#include <vm/vm_object.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <vm/vm_kern.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
#include <vm/vm_extern.h>
|
2002-07-07 23:05:27 +00:00
|
|
|
#include <vm/vm_pager.h>
|
2002-12-15 19:17:57 +00:00
|
|
|
#include <vm/swap_pager.h>
|
1995-12-07 12:48:31 +00:00
|
|
|
|
2014-10-04 18:38:14 +00:00
|
|
|
#include <machine/cpu.h>
|
|
|
|
|
2002-06-22 01:26:02 +00:00
|
|
|
/*
|
|
|
|
* MPSAFE
|
2003-01-20 17:46:48 +00:00
|
|
|
*
|
|
|
|
* WARNING! This code calls vm_map_check_protection() which only checks
|
|
|
|
* the associated vm_map_entry range. It does not determine whether the
|
|
|
|
* contents of the memory is actually readable or writable. In most cases
|
|
|
|
* just checking the vm_map_entry is sufficient within the kernel's address
|
|
|
|
* space.
|
2002-06-22 01:26:02 +00:00
|
|
|
*/
|
1994-05-24 10:09:53 +00:00
|
|
|
int
|
2018-01-12 03:50:44 +00:00
|
|
|
kernacc(void *addr, int len, int rw)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
|
|
|
boolean_t rv;
|
|
|
|
vm_offset_t saddr, eaddr;
|
1999-10-30 06:32:05 +00:00
|
|
|
vm_prot_t prot;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2002-01-17 16:46:26 +00:00
|
|
|
KASSERT((rw & ~VM_PROT_ALL) == 0,
|
1999-10-30 06:32:05 +00:00
|
|
|
("illegal ``rw'' argument to kernacc (%x)\n", rw));
|
2005-01-22 19:21:29 +00:00
|
|
|
|
2018-08-29 12:24:19 +00:00
|
|
|
if ((vm_offset_t)addr + len > vm_map_max(kernel_map) ||
|
2005-01-22 19:21:29 +00:00
|
|
|
(vm_offset_t)addr + len < (vm_offset_t)addr)
|
|
|
|
return (FALSE);
|
|
|
|
|
1999-10-30 06:32:05 +00:00
|
|
|
prot = rw;
|
1998-10-13 08:24:45 +00:00
|
|
|
saddr = trunc_page((vm_offset_t)addr);
|
|
|
|
eaddr = round_page((vm_offset_t)addr + len);
|
2003-11-10 01:37:40 +00:00
|
|
|
vm_map_lock_read(kernel_map);
|
1994-05-24 10:09:53 +00:00
|
|
|
rv = vm_map_check_protection(kernel_map, saddr, eaddr, prot);
|
2003-11-10 01:37:40 +00:00
|
|
|
vm_map_unlock_read(kernel_map);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
return (rv == TRUE);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
2002-06-22 01:26:02 +00:00
|
|
|
/*
|
|
|
|
* MPSAFE
|
2003-01-20 17:46:48 +00:00
|
|
|
*
|
|
|
|
* WARNING! This code calls vm_map_check_protection() which only checks
|
|
|
|
* the associated vm_map_entry range. It does not determine whether the
|
|
|
|
* contents of the memory is actually readable or writable. vmapbuf(),
|
|
|
|
* vm_fault_quick(), or copyin()/copout()/su*()/fu*() functions should be
|
2016-05-02 20:16:29 +00:00
|
|
|
* used in conjunction with this call.
|
2002-06-22 01:26:02 +00:00
|
|
|
*/
|
1994-05-24 10:09:53 +00:00
|
|
|
int
|
2018-01-12 03:50:44 +00:00
|
|
|
useracc(void *addr, int len, int rw)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
|
|
|
boolean_t rv;
|
1999-10-30 06:32:05 +00:00
|
|
|
vm_prot_t prot;
|
2002-09-21 22:07:17 +00:00
|
|
|
vm_map_t map;
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2002-01-17 16:46:26 +00:00
|
|
|
KASSERT((rw & ~VM_PROT_ALL) == 0,
|
1999-10-30 06:32:05 +00:00
|
|
|
("illegal ``rw'' argument to useracc (%x)\n", rw));
|
|
|
|
prot = rw;
|
2002-09-21 22:07:17 +00:00
|
|
|
map = &curproc->p_vmspace->vm_map;
|
|
|
|
if ((vm_offset_t)addr + len > vm_map_max(map) ||
|
|
|
|
(vm_offset_t)addr + len < (vm_offset_t)addr) {
|
1994-05-25 09:21:21 +00:00
|
|
|
return (FALSE);
|
|
|
|
}
|
2003-11-10 01:37:40 +00:00
|
|
|
vm_map_lock_read(map);
|
2002-09-21 22:07:17 +00:00
|
|
|
rv = vm_map_check_protection(map, trunc_page((vm_offset_t)addr),
|
|
|
|
round_page((vm_offset_t)addr + len), prot);
|
2003-11-10 01:37:40 +00:00
|
|
|
vm_map_unlock_read(map);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
return (rv == TRUE);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
|
|
|
|
2004-03-05 22:03:11 +00:00
|
|
|
int
|
2004-03-15 06:43:51 +00:00
|
|
|
vslock(void *addr, size_t len)
|
2004-03-05 22:03:11 +00:00
|
|
|
{
|
2004-03-15 09:11:23 +00:00
|
|
|
vm_offset_t end, last, start;
|
|
|
|
vm_size_t npages;
|
|
|
|
int error;
|
2004-03-05 22:03:11 +00:00
|
|
|
|
2004-03-15 09:11:23 +00:00
|
|
|
last = (vm_offset_t)addr + len;
|
2004-03-15 06:42:40 +00:00
|
|
|
start = trunc_page((vm_offset_t)addr);
|
2004-03-15 09:11:23 +00:00
|
|
|
end = round_page(last);
|
|
|
|
if (last < (vm_offset_t)addr || end < (vm_offset_t)addr)
|
2004-03-05 22:03:11 +00:00
|
|
|
return (EINVAL);
|
|
|
|
npages = atop(end - start);
|
Provide separate accounting for user-wired pages.
Historically we have not distinguished between kernel wirings and user
wirings for accounting purposes. User wirings (via mlock(2)) were
subject to a global limit on the number of wired pages, so if large
swaths of physical memory were wired by the kernel, as happens with
the ZFS ARC among other things, the limit could be exceeded, causing
user wirings to fail.
The change adds a new counter, v_user_wire_count, which counts the
number of virtual pages wired by user processes via mlock(2) and
mlockall(2). Only user-wired pages are subject to the system-wide
limit which helps provide some safety against deadlocks. In
particular, while sources of kernel wirings typically support some
backpressure mechanism, there is no way to reclaim user-wired pages
shorting of killing the wiring process. The limit is exported as
vm.max_user_wired, renamed from vm.max_wired, and changed from u_int
to u_long.
The choice to count virtual user-wired pages rather than physical
pages was done for simplicity. There are mechanisms that can cause
user-wired mappings to be destroyed while maintaining a wiring of
the backing physical page; these make it difficult to accurately
track user wirings at the physical page layer.
The change also closes some holes which allowed user wirings to succeed
even when they would cause the system limit to be exceeded. For
instance, mmap() may now fail with ENOMEM in a process that has called
mlockall(MCL_FUTURE) if the new mapping would cause the user wiring
limit to be exceeded.
Note that bhyve -S is subject to the user wiring limit, which defaults
to 1/3 of physical RAM. Users that wish to exceed the limit must tune
vm.max_user_wired.
Reviewed by: kib, ngie (mlock() test changes)
Tested by: pho (earlier version)
MFC after: 45 days
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D19908
2019-05-13 16:38:48 +00:00
|
|
|
if (npages > vm_page_max_user_wired)
|
2004-03-05 22:03:11 +00:00
|
|
|
return (ENOMEM);
|
2004-03-15 06:42:40 +00:00
|
|
|
error = vm_map_wire(&curproc->p_vmspace->vm_map, start, end,
|
2004-05-07 11:43:24 +00:00
|
|
|
VM_MAP_WIRE_SYSTEM | VM_MAP_WIRE_NOHOLES);
|
2018-03-24 13:51:27 +00:00
|
|
|
if (error == KERN_SUCCESS) {
|
|
|
|
curthread->td_vslock_sz += len;
|
|
|
|
return (0);
|
|
|
|
}
|
|
|
|
|
2004-03-15 06:42:40 +00:00
|
|
|
/*
|
|
|
|
* Return EFAULT on error to match copy{in,out}() behaviour
|
|
|
|
* rather than returning ENOMEM like mlock() would.
|
|
|
|
*/
|
2018-03-24 13:51:27 +00:00
|
|
|
return (EFAULT);
|
2004-03-05 22:03:11 +00:00
|
|
|
}
|
|
|
|
|
2004-03-15 06:42:40 +00:00
|
|
|
void
|
2004-03-15 06:43:51 +00:00
|
|
|
vsunlock(void *addr, size_t len)
|
2004-03-05 22:03:11 +00:00
|
|
|
{
|
|
|
|
|
2004-03-15 06:42:40 +00:00
|
|
|
/* Rely on the parameter sanity checks performed by vslock(). */
|
2018-03-24 13:51:27 +00:00
|
|
|
MPASS(curthread->td_vslock_sz >= len);
|
|
|
|
curthread->td_vslock_sz -= len;
|
2004-03-15 06:42:40 +00:00
|
|
|
(void)vm_map_unwire(&curproc->p_vmspace->vm_map,
|
|
|
|
trunc_page((vm_offset_t)addr), round_page((vm_offset_t)addr + len),
|
2004-03-15 06:43:51 +00:00
|
|
|
VM_MAP_WIRE_SYSTEM | VM_MAP_WIRE_NOHOLES);
|
2004-03-05 22:03:11 +00:00
|
|
|
}
|
|
|
|
|
2005-12-16 18:34:14 +00:00
|
|
|
/*
|
|
|
|
* Pin the page contained within the given object at the given offset. If the
|
|
|
|
* page is not resident, allocate and load it using the given object's pager.
|
|
|
|
* Return the pinned page if successful; otherwise, return NULL.
|
|
|
|
*/
|
|
|
|
static vm_page_t
|
2013-08-05 08:55:35 +00:00
|
|
|
vm_imgact_hold_page(vm_object_t object, vm_ooffset_t offset)
|
2005-12-16 18:34:14 +00:00
|
|
|
{
|
2015-06-12 11:32:20 +00:00
|
|
|
vm_page_t m;
|
2005-12-16 18:34:14 +00:00
|
|
|
vm_pindex_t pindex;
|
|
|
|
|
|
|
|
pindex = OFF_TO_IDX(offset);
|
2020-02-28 20:34:30 +00:00
|
|
|
(void)vm_page_grab_valid_unlocked(&m, object, pindex,
|
2019-09-10 19:08:01 +00:00
|
|
|
VM_ALLOC_NORMAL | VM_ALLOC_NOBUSY | VM_ALLOC_WIRED);
|
2005-12-16 18:34:14 +00:00
|
|
|
return (m);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Return a CPU private mapping to the page at the given offset within the
|
|
|
|
* given object. The page is pinned before it is mapped.
|
|
|
|
*/
|
|
|
|
struct sf_buf *
|
|
|
|
vm_imgact_map_page(vm_object_t object, vm_ooffset_t offset)
|
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
2013-08-05 08:55:35 +00:00
|
|
|
m = vm_imgact_hold_page(object, offset);
|
2005-12-16 18:34:14 +00:00
|
|
|
if (m == NULL)
|
|
|
|
return (NULL);
|
|
|
|
sched_pin();
|
|
|
|
return (sf_buf_alloc(m, SFB_CPUPRIVATE));
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Destroy the given CPU private mapping and unpin the page that it mapped.
|
|
|
|
*/
|
|
|
|
void
|
2013-08-05 08:55:35 +00:00
|
|
|
vm_imgact_unmap_page(struct sf_buf *sf)
|
2005-12-16 18:34:14 +00:00
|
|
|
{
|
|
|
|
vm_page_t m;
|
|
|
|
|
|
|
|
m = sf_buf_page(sf);
|
|
|
|
sf_buf_free(sf);
|
|
|
|
sched_unpin();
|
2019-07-08 19:46:20 +00:00
|
|
|
vm_page_unwire(m, PQ_ACTIVE);
|
2005-12-16 18:34:14 +00:00
|
|
|
}
|
|
|
|
|
2009-10-21 18:38:02 +00:00
|
|
|
void
|
|
|
|
vm_sync_icache(vm_map_t map, vm_offset_t va, vm_offset_t sz)
|
|
|
|
{
|
|
|
|
|
|
|
|
pmap_sync_icache(map->pmap, va, sz);
|
|
|
|
}
|
|
|
|
|
2020-04-26 20:08:57 +00:00
|
|
|
vm_object_t kstack_object;
|
2019-08-06 23:15:34 +00:00
|
|
|
static uma_zone_t kstack_cache;
|
2019-11-22 16:30:47 +00:00
|
|
|
static int kstack_cache_size;
|
2012-10-26 17:31:35 +00:00
|
|
|
|
2019-08-06 23:15:34 +00:00
|
|
|
static int
|
|
|
|
sysctl_kstack_cache_size(SYSCTL_HANDLER_ARGS)
|
|
|
|
{
|
2019-12-07 17:28:41 +00:00
|
|
|
int error, oldsize;
|
2019-08-06 23:15:34 +00:00
|
|
|
|
2019-12-07 17:28:41 +00:00
|
|
|
oldsize = kstack_cache_size;
|
|
|
|
error = sysctl_handle_int(oidp, arg1, arg2, req);
|
|
|
|
if (error == 0 && req->newptr && oldsize != kstack_cache_size)
|
|
|
|
uma_zone_set_maxcache(kstack_cache, kstack_cache_size);
|
2019-08-06 23:15:34 +00:00
|
|
|
return (error);
|
|
|
|
}
|
2020-01-12 05:08:57 +00:00
|
|
|
SYSCTL_PROC(_vm, OID_AUTO, kstack_cache_size,
|
|
|
|
CTLTYPE_INT|CTLFLAG_MPSAFE|CTLFLAG_RW, &kstack_cache_size, 0,
|
|
|
|
sysctl_kstack_cache_size, "IU", "Maximum number of cached kernel stacks");
|
2009-09-01 11:41:51 +00:00
|
|
|
|
2003-06-14 23:23:55 +00:00
|
|
|
/*
|
|
|
|
* Create the kernel stack (including pcb for i386) for a new thread.
|
|
|
|
*/
|
2019-08-06 23:15:34 +00:00
|
|
|
static vm_offset_t
|
2020-04-26 20:08:57 +00:00
|
|
|
vm_thread_stack_create(struct domainset *ds, int pages)
|
2003-06-14 23:23:55 +00:00
|
|
|
{
|
2019-08-06 23:15:34 +00:00
|
|
|
vm_page_t ma[KSTACK_MAX_PAGES];
|
2003-06-14 23:23:55 +00:00
|
|
|
vm_offset_t ks;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Get a kernel virtual address for this thread's kstack.
|
|
|
|
*/
|
2013-08-07 06:21:20 +00:00
|
|
|
ks = kva_alloc((pages + KSTACK_GUARD_PAGES) * PAGE_SIZE);
|
2007-11-05 11:36:16 +00:00
|
|
|
if (ks == 0) {
|
2019-09-19 07:28:24 +00:00
|
|
|
printf("%s: kstack allocation failed\n", __func__);
|
2007-11-05 11:36:16 +00:00
|
|
|
return (0);
|
|
|
|
}
|
2018-10-22 17:04:04 +00:00
|
|
|
|
2003-06-14 23:23:55 +00:00
|
|
|
if (KSTACK_GUARD_PAGES != 0) {
|
|
|
|
pmap_qremove(ks, KSTACK_GUARD_PAGES);
|
|
|
|
ks += KSTACK_GUARD_PAGES * PAGE_SIZE;
|
|
|
|
}
|
2019-08-06 23:15:34 +00:00
|
|
|
|
2020-04-26 20:08:57 +00:00
|
|
|
/*
|
|
|
|
* Allocate physical pages to back the stack.
|
2003-06-14 23:23:55 +00:00
|
|
|
*/
|
2020-04-26 20:08:57 +00:00
|
|
|
vm_thread_stack_back(ds, ks, ma, pages, VM_ALLOC_NORMAL);
|
2020-02-19 09:10:11 +00:00
|
|
|
for (i = 0; i < pages; i++)
|
2020-01-23 04:54:49 +00:00
|
|
|
vm_page_valid(ma[i]);
|
2003-06-14 23:23:55 +00:00
|
|
|
pmap_qenter(ks, ma, pages);
|
2019-08-06 23:15:34 +00:00
|
|
|
|
|
|
|
return (ks);
|
2003-06-14 23:23:55 +00:00
|
|
|
}
|
|
|
|
|
2009-09-01 11:41:51 +00:00
|
|
|
static void
|
2020-04-26 20:08:57 +00:00
|
|
|
vm_thread_stack_dispose(vm_offset_t ks, int pages)
|
2003-06-14 23:23:55 +00:00
|
|
|
{
|
|
|
|
vm_page_t m;
|
2020-04-26 20:08:57 +00:00
|
|
|
vm_pindex_t pindex;
|
2009-09-01 11:41:51 +00:00
|
|
|
int i;
|
2003-06-14 23:23:55 +00:00
|
|
|
|
2020-04-26 20:08:57 +00:00
|
|
|
pindex = atop(ks - VM_MIN_KERNEL_ADDRESS);
|
|
|
|
|
2003-06-14 23:23:55 +00:00
|
|
|
pmap_qremove(ks, pages);
|
2020-04-26 20:08:57 +00:00
|
|
|
VM_OBJECT_WLOCK(kstack_object);
|
2003-06-14 23:23:55 +00:00
|
|
|
for (i = 0; i < pages; i++) {
|
2020-04-26 20:08:57 +00:00
|
|
|
m = vm_page_lookup(kstack_object, pindex + i);
|
2003-06-14 23:23:55 +00:00
|
|
|
if (m == NULL)
|
2019-09-19 07:28:24 +00:00
|
|
|
panic("%s: kstack already missing?", __func__);
|
2020-02-19 09:10:11 +00:00
|
|
|
vm_page_xbusy_claim(m);
|
2019-06-07 18:23:29 +00:00
|
|
|
vm_page_unwire_noq(m);
|
2003-06-14 23:23:55 +00:00
|
|
|
vm_page_free(m);
|
|
|
|
}
|
2020-04-26 20:08:57 +00:00
|
|
|
VM_OBJECT_WUNLOCK(kstack_object);
|
2021-04-13 21:39:55 +00:00
|
|
|
kasan_mark((void *)ks, ptoa(pages), ptoa(pages), 0);
|
2013-08-07 06:21:20 +00:00
|
|
|
kva_free(ks - (KSTACK_GUARD_PAGES * PAGE_SIZE),
|
2003-06-14 23:23:55 +00:00
|
|
|
(pages + KSTACK_GUARD_PAGES) * PAGE_SIZE);
|
2009-09-01 11:41:51 +00:00
|
|
|
}
|
|
|
|
|
2019-08-06 23:15:34 +00:00
|
|
|
/*
|
|
|
|
* Allocate the kernel stack for a new thread.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
vm_thread_new(struct thread *td, int pages)
|
|
|
|
{
|
|
|
|
vm_offset_t ks;
|
|
|
|
|
|
|
|
/* Bounds check */
|
|
|
|
if (pages <= 1)
|
|
|
|
pages = kstack_pages;
|
|
|
|
else if (pages > KSTACK_MAX_PAGES)
|
|
|
|
pages = KSTACK_MAX_PAGES;
|
|
|
|
|
|
|
|
ks = 0;
|
2020-04-26 20:08:57 +00:00
|
|
|
if (pages == kstack_pages && kstack_cache != NULL)
|
2019-08-06 23:15:34 +00:00
|
|
|
ks = (vm_offset_t)uma_zalloc(kstack_cache, M_NOWAIT);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Ensure that kstack objects can draw pages from any memory
|
|
|
|
* domain. Otherwise a local memory shortage can block a process
|
|
|
|
* swap-in.
|
|
|
|
*/
|
|
|
|
if (ks == 0)
|
|
|
|
ks = vm_thread_stack_create(DOMAINSET_PREF(PCPU_GET(domain)),
|
2020-04-26 20:08:57 +00:00
|
|
|
pages);
|
2019-08-06 23:15:34 +00:00
|
|
|
if (ks == 0)
|
|
|
|
return (0);
|
|
|
|
td->td_kstack = ks;
|
|
|
|
td->td_kstack_pages = pages;
|
2021-04-13 21:39:55 +00:00
|
|
|
kasan_mark((void *)ks, ptoa(pages), ptoa(pages), 0);
|
2021-08-10 21:15:03 +00:00
|
|
|
kmsan_mark((void *)ks, ptoa(pages), KMSAN_STATE_UNINIT);
|
2019-08-06 23:15:34 +00:00
|
|
|
return (1);
|
|
|
|
}
|
|
|
|
|
2009-09-01 11:41:51 +00:00
|
|
|
/*
|
|
|
|
* Dispose of a thread's kernel stack.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_thread_dispose(struct thread *td)
|
|
|
|
{
|
|
|
|
vm_offset_t ks;
|
|
|
|
int pages;
|
|
|
|
|
|
|
|
pages = td->td_kstack_pages;
|
|
|
|
ks = td->td_kstack;
|
2007-11-05 11:36:16 +00:00
|
|
|
td->td_kstack = 0;
|
2009-09-01 11:41:51 +00:00
|
|
|
td->td_kstack_pages = 0;
|
2021-04-13 21:39:55 +00:00
|
|
|
kasan_mark((void *)ks, 0, ptoa(pages), KASAN_KSTACK_FREED);
|
2019-08-06 23:15:34 +00:00
|
|
|
if (pages == kstack_pages)
|
|
|
|
uma_zfree(kstack_cache, (void *)ks);
|
|
|
|
else
|
2020-04-26 20:08:57 +00:00
|
|
|
vm_thread_stack_dispose(ks, pages);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Allocate physical pages, following the specified NUMA policy, to back a
|
|
|
|
* kernel stack.
|
|
|
|
*/
|
|
|
|
void
|
|
|
|
vm_thread_stack_back(struct domainset *ds, vm_offset_t ks, vm_page_t ma[],
|
|
|
|
int npages, int req_class)
|
|
|
|
{
|
|
|
|
vm_pindex_t pindex;
|
|
|
|
int n;
|
|
|
|
|
|
|
|
pindex = atop(ks - VM_MIN_KERNEL_ADDRESS);
|
|
|
|
|
|
|
|
VM_OBJECT_WLOCK(kstack_object);
|
|
|
|
for (n = 0; n < npages;) {
|
|
|
|
if (vm_ndomains > 1)
|
|
|
|
kstack_object->domain.dr_policy = ds;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Use WAITFAIL to force a reset of the domain selection policy
|
|
|
|
* if we had to sleep for pages.
|
|
|
|
*/
|
|
|
|
n += vm_page_grab_pages(kstack_object, pindex + n,
|
|
|
|
req_class | VM_ALLOC_WIRED | VM_ALLOC_WAITFAIL,
|
|
|
|
&ma[n], npages - n);
|
|
|
|
}
|
|
|
|
VM_OBJECT_WUNLOCK(kstack_object);
|
2009-09-01 11:41:51 +00:00
|
|
|
}
|
|
|
|
|
2019-08-06 23:15:34 +00:00
|
|
|
static int
|
|
|
|
kstack_import(void *arg, void **store, int cnt, int domain, int flags)
|
2009-09-01 11:41:51 +00:00
|
|
|
{
|
2019-08-25 21:14:46 +00:00
|
|
|
struct domainset *ds;
|
2019-08-06 23:15:34 +00:00
|
|
|
int i;
|
2009-09-01 11:41:51 +00:00
|
|
|
|
2019-08-25 21:14:46 +00:00
|
|
|
if (domain == UMA_ANYDOMAIN)
|
|
|
|
ds = DOMAINSET_RR();
|
|
|
|
else
|
|
|
|
ds = DOMAINSET_PREF(domain);
|
|
|
|
|
2019-08-06 23:15:34 +00:00
|
|
|
for (i = 0; i < cnt; i++) {
|
2020-04-26 20:08:57 +00:00
|
|
|
store[i] = (void *)vm_thread_stack_create(ds, kstack_pages);
|
2019-08-06 23:15:34 +00:00
|
|
|
if (store[i] == NULL)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
return (i);
|
|
|
|
}
|
2009-09-01 11:41:51 +00:00
|
|
|
|
2019-08-06 23:15:34 +00:00
|
|
|
static void
|
|
|
|
kstack_release(void *arg, void **store, int cnt)
|
|
|
|
{
|
|
|
|
vm_offset_t ks;
|
|
|
|
int i;
|
2009-09-01 11:41:51 +00:00
|
|
|
|
2019-08-06 23:15:34 +00:00
|
|
|
for (i = 0; i < cnt; i++) {
|
|
|
|
ks = (vm_offset_t)store[i];
|
2020-04-26 20:08:57 +00:00
|
|
|
vm_thread_stack_dispose(ks, kstack_pages);
|
2009-09-01 11:41:51 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
2019-08-06 23:15:34 +00:00
|
|
|
kstack_cache_init(void *null)
|
2009-09-01 11:41:51 +00:00
|
|
|
{
|
2020-04-26 20:08:57 +00:00
|
|
|
kstack_object = vm_object_allocate(OBJT_SWAP,
|
|
|
|
atop(VM_MAX_KERNEL_ADDRESS - VM_MIN_KERNEL_ADDRESS));
|
2019-08-06 23:15:34 +00:00
|
|
|
kstack_cache = uma_zcache_create("kstack_cache",
|
|
|
|
kstack_pages * PAGE_SIZE, NULL, NULL, NULL, NULL,
|
|
|
|
kstack_import, kstack_release, NULL,
|
2020-01-04 18:48:13 +00:00
|
|
|
UMA_ZONE_FIRSTTOUCH);
|
2019-11-22 16:30:47 +00:00
|
|
|
kstack_cache_size = imax(128, mp_ncpus * 4);
|
2019-08-06 23:15:34 +00:00
|
|
|
uma_zone_set_maxcache(kstack_cache, kstack_cache_size);
|
2003-06-14 23:23:55 +00:00
|
|
|
}
|
2020-04-26 20:08:57 +00:00
|
|
|
SYSINIT(vm_kstacks, SI_SUB_KMEM, SI_ORDER_ANY, kstack_cache_init, NULL);
|
2009-09-01 11:41:51 +00:00
|
|
|
|
2014-10-04 18:38:14 +00:00
|
|
|
#ifdef KSTACK_USAGE_PROF
|
|
|
|
/*
|
|
|
|
* Track maximum stack used by a thread in kernel.
|
|
|
|
*/
|
|
|
|
static int max_kstack_used;
|
|
|
|
|
|
|
|
SYSCTL_INT(_debug, OID_AUTO, max_kstack_used, CTLFLAG_RD,
|
|
|
|
&max_kstack_used, 0,
|
2021-11-03 19:49:24 +00:00
|
|
|
"Maximum stack depth used by a thread in kernel");
|
2014-10-04 18:38:14 +00:00
|
|
|
|
|
|
|
void
|
|
|
|
intr_prof_stack_use(struct thread *td, struct trapframe *frame)
|
|
|
|
{
|
|
|
|
vm_offset_t stack_top;
|
|
|
|
vm_offset_t current;
|
|
|
|
int used, prev_used;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Testing for interrupted kernel mode isn't strictly
|
|
|
|
* needed. It optimizes the execution, since interrupts from
|
|
|
|
* usermode will have only the trap frame on the stack.
|
|
|
|
*/
|
|
|
|
if (TRAPF_USERMODE(frame))
|
|
|
|
return;
|
|
|
|
|
|
|
|
stack_top = td->td_kstack + td->td_kstack_pages * PAGE_SIZE;
|
|
|
|
current = (vm_offset_t)(uintptr_t)&stack_top;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Try to detect if interrupt is using kernel thread stack.
|
|
|
|
* Hardware could use a dedicated stack for interrupt handling.
|
|
|
|
*/
|
|
|
|
if (stack_top <= current || current < td->td_kstack)
|
|
|
|
return;
|
|
|
|
|
|
|
|
used = stack_top - current;
|
|
|
|
for (;;) {
|
|
|
|
prev_used = max_kstack_used;
|
|
|
|
if (prev_used >= used)
|
|
|
|
break;
|
|
|
|
if (atomic_cmpset_int(&max_kstack_used, prev_used, used))
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#endif /* KSTACK_USAGE_PROF */
|
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
* Implement fork's actions on an address space.
|
|
|
|
* Here we arrange for the address space to be copied or referenced,
|
|
|
|
* allocate a user struct (pcb and kernel stack), then call the
|
|
|
|
* machine-dependent layer to fill those in and make the new process
|
1997-04-07 07:16:06 +00:00
|
|
|
* ready to run. The new process is set up so that it returns directly
|
|
|
|
* to user mode to avoid stack copying and relocation problems.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2007-11-05 11:36:16 +00:00
|
|
|
int
|
2018-01-12 03:50:44 +00:00
|
|
|
vm_forkproc(struct thread *td, struct proc *p2, struct thread *td2,
|
|
|
|
struct vmspace *vm2, int flags)
|
1994-05-24 10:09:53 +00:00
|
|
|
{
|
2001-09-12 08:38:13 +00:00
|
|
|
struct proc *p1 = td->td_proc;
|
2018-09-06 19:28:52 +00:00
|
|
|
struct domainset *dset;
|
2007-11-05 11:36:16 +00:00
|
|
|
int error;
|
1994-05-25 09:21:21 +00:00
|
|
|
|
1999-12-06 04:53:08 +00:00
|
|
|
if ((flags & RFPROC) == 0) {
|
|
|
|
/*
|
|
|
|
* Divorce the memory, if it is shared, essentially
|
|
|
|
* this changes shared memory amongst threads, into
|
|
|
|
* COW locally.
|
|
|
|
*/
|
|
|
|
if ((flags & RFMEM) == 0) {
|
2021-05-13 12:33:23 +00:00
|
|
|
error = vmspace_unshare(p1);
|
|
|
|
if (error)
|
|
|
|
return (error);
|
1999-12-06 04:53:08 +00:00
|
|
|
}
|
2002-02-07 20:58:47 +00:00
|
|
|
cpu_fork(td, p2, td2, flags);
|
2007-11-05 11:36:16 +00:00
|
|
|
return (0);
|
1999-12-06 04:53:08 +00:00
|
|
|
}
|
|
|
|
|
1997-04-13 01:48:35 +00:00
|
|
|
if (flags & RFMEM) {
|
|
|
|
p2->p_vmspace = p1->p_vmspace;
|
2020-11-04 16:30:56 +00:00
|
|
|
refcount_acquire(&p1->p_vmspace->vm_refcnt);
|
1997-04-13 01:48:35 +00:00
|
|
|
}
|
2018-09-06 19:28:52 +00:00
|
|
|
dset = td2->td_domain.dr_policy;
|
|
|
|
while (vm_page_count_severe_set(&dset->ds_mask)) {
|
2020-09-08 23:28:09 +00:00
|
|
|
vm_wait_doms(&dset->ds_mask, 0);
|
These changes embody the support of the fully coherent merged VM buffer cache,
much higher filesystem I/O performance, and much better paging performance. It
represents the culmination of over 6 months of R&D.
The majority of the merged VM/cache work is by John Dyson.
The following highlights the most significant changes. Additionally, there are
(mostly minor) changes to the various filesystem modules (nfs, msdosfs, etc) to
support the new VM/buffer scheme.
vfs_bio.c:
Significant rewrite of most of vfs_bio to support the merged VM buffer cache
scheme. The scheme is almost fully compatible with the old filesystem
interface. Significant improvement in the number of opportunities for write
clustering.
vfs_cluster.c, vfs_subr.c
Upgrade and performance enhancements in vfs layer code to support merged
VM/buffer cache. Fixup of vfs_cluster to eliminate the bogus pagemove stuff.
vm_object.c:
Yet more improvements in the collapse code. Elimination of some windows that
can cause list corruption.
vm_pageout.c:
Fixed it, it really works better now. Somehow in 2.0, some "enhancements"
broke the code. This code has been reworked from the ground-up.
vm_fault.c, vm_page.c, pmap.c, vm_object.c
Support for small-block filesystems with merged VM/buffer cache scheme.
pmap.c vm_map.c
Dynamic kernel VM size, now we dont have to pre-allocate excessive numbers of
kernel PTs.
vm_glue.c
Much simpler and more effective swapping code. No more gratuitous swapping.
proc.h
Fixed the problem that the p_lock flag was not being cleared on a fork.
swap_pager.c, vnode_pager.c
Removal of old vfs_bio cruft to support the past pseudo-coherency. Now the
code doesn't need it anymore.
machdep.c
Changes to better support the parameter values for the merged VM/buffer cache
scheme.
machdep.c, kern_exec.c, vm_glue.c
Implemented a seperate submap for temporary exec string space and another one
to contain process upages. This eliminates all map fragmentation problems
that previously existed.
ffs_inode.c, ufs_inode.c, ufs_readwrite.c
Changes for merged VM/buffer cache. Add "bypass" support for sneaking in on
busy buffers.
Submitted by: John Dyson and David Greenman
1995-01-09 16:06:02 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
|
1997-04-13 01:48:35 +00:00
|
|
|
if ((flags & RFMEM) == 0) {
|
2007-11-05 11:36:16 +00:00
|
|
|
p2->p_vmspace = vm2;
|
1997-04-07 07:16:06 +00:00
|
|
|
if (p1->p_vmspace->vm_shm)
|
|
|
|
shmfork(p1, p2);
|
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
|
|
|
|
/*
|
1997-04-07 07:16:06 +00:00
|
|
|
* cpu_fork will copy and update the pcb, set up the kernel stack,
|
|
|
|
* and make the child ready to run.
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
2002-02-07 20:58:47 +00:00
|
|
|
cpu_fork(td, p2, td2, flags);
|
2007-11-05 11:36:16 +00:00
|
|
|
return (0);
|
1994-05-24 10:09:53 +00:00
|
|
|
}
|
2001-09-10 04:28:58 +00:00
|
|
|
|
|
|
|
/*
|
2016-05-02 20:16:29 +00:00
|
|
|
* Called after process has been wait(2)'ed upon and is being reaped.
|
2001-09-10 04:28:58 +00:00
|
|
|
* The idea is to reclaim resources that we could not reclaim while
|
|
|
|
* the process was still executing.
|
|
|
|
*/
|
|
|
|
void
|
2023-02-13 18:23:21 +00:00
|
|
|
vm_waitproc(struct proc *p)
|
2001-09-10 04:28:58 +00:00
|
|
|
{
|
|
|
|
|
2002-02-05 21:23:05 +00:00
|
|
|
vmspace_exitfree(p); /* and clean-out the vmspace */
|
2001-09-10 04:28:58 +00:00
|
|
|
}
|
1994-05-24 10:09:53 +00:00
|
|
|
|
If a thread that is swapped out is made runnable, then the setrunnable()
routine wakes up proc0 so that proc0 can swap the thread back in.
Historically, this has been done by waking up proc0 directly from
setrunnable() itself via a wakeup(). When waking up a sleeping thread
that was swapped out (the usual case when waking proc0 since only sleeping
threads are eligible to be swapped out), this resulted in a bit of
recursion (e.g. wakeup() -> setrunnable() -> wakeup()).
With sleep queues having separate locks in 6.x and later, this caused a
spin lock LOR (sleepq lock -> sched_lock/thread lock -> sleepq lock).
An attempt was made to fix this in 7.0 by making the proc0 wakeup use
the ithread mechanism for doing the wakeup. However, this required
grabbing proc0's thread lock to perform the wakeup. If proc0 was asleep
elsewhere in the kernel (e.g. waiting for disk I/O), then this degenerated
into the same LOR since the thread lock would be some other sleepq lock.
Fix this by deferring the wakeup of the swapper until after the sleepq
lock held by the upper layer has been locked. The setrunnable() routine
now returns a boolean value to indicate whether or not proc0 needs to be
woken up. The end result is that consumers of the sleepq API such as
*sleep/wakeup, condition variables, sx locks, and lockmgr, have to wakeup
proc0 if they get a non-zero return value from sleepq_abort(),
sleepq_broadcast(), or sleepq_signal().
Discussed with: jeff
Glanced at by: sam
Tested by: Jurgen Weber jurgen - ish com au
MFC after: 2 weeks
2008-08-05 20:02:31 +00:00
|
|
|
void
|
|
|
|
kick_proc0(void)
|
2005-05-23 23:01:53 +00:00
|
|
|
{
|
|
|
|
|
If a thread that is swapped out is made runnable, then the setrunnable()
routine wakes up proc0 so that proc0 can swap the thread back in.
Historically, this has been done by waking up proc0 directly from
setrunnable() itself via a wakeup(). When waking up a sleeping thread
that was swapped out (the usual case when waking proc0 since only sleeping
threads are eligible to be swapped out), this resulted in a bit of
recursion (e.g. wakeup() -> setrunnable() -> wakeup()).
With sleep queues having separate locks in 6.x and later, this caused a
spin lock LOR (sleepq lock -> sched_lock/thread lock -> sleepq lock).
An attempt was made to fix this in 7.0 by making the proc0 wakeup use
the ithread mechanism for doing the wakeup. However, this required
grabbing proc0's thread lock to perform the wakeup. If proc0 was asleep
elsewhere in the kernel (e.g. waiting for disk I/O), then this degenerated
into the same LOR since the thread lock would be some other sleepq lock.
Fix this by deferring the wakeup of the swapper until after the sleepq
lock held by the upper layer has been locked. The setrunnable() routine
now returns a boolean value to indicate whether or not proc0 needs to be
woken up. The end result is that consumers of the sleepq API such as
*sleep/wakeup, condition variables, sx locks, and lockmgr, have to wakeup
proc0 if they get a non-zero return value from sleepq_abort(),
sleepq_broadcast(), or sleepq_signal().
Discussed with: jeff
Glanced at by: sam
Tested by: Jurgen Weber jurgen - ish com au
MFC after: 2 weeks
2008-08-05 20:02:31 +00:00
|
|
|
wakeup(&proc0);
|
|
|
|
}
|