2005-01-06 23:35:40 +00:00
|
|
|
/*-
|
2017-11-20 19:43:44 +00:00
|
|
|
* SPDX-License-Identifier: BSD-3-Clause
|
|
|
|
|
*
|
1994-05-24 10:09:53 +00:00
|
|
|
* Copyright (c) 1980, 1986, 1989, 1993
|
|
|
|
|
* The Regents of the University of California. All rights reserved.
|
|
|
|
|
* (c) UNIX System Laboratories, Inc.
|
|
|
|
|
* All or some portions of this file are derived from material licensed
|
|
|
|
|
* to the University of California by American Telephone and Telegraph
|
|
|
|
|
* Co. or Unix System Laboratories, Inc. and are reproduced herein with
|
|
|
|
|
* the permission of UNIX System Laboratories, Inc.
|
|
|
|
|
*
|
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
2016-09-15 13:16:20 +00:00
|
|
|
* 3. Neither the name of the University nor the names of its contributors
|
1994-05-24 10:09:53 +00:00
|
|
|
* may be used to endorse or promote products derived from this software
|
|
|
|
|
* without specific prior written permission.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
|
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
|
*
|
1996-03-11 05:52:50 +00:00
|
|
|
* @(#)param.c 8.3 (Berkeley) 8/20/94
|
1994-05-24 10:09:53 +00:00
|
|
|
*/
|
|
|
|
|
|
2003-06-11 00:56:59 +00:00
|
|
|
#include <sys/cdefs.h>
|
|
|
|
|
__FBSDID("$FreeBSD$");
|
|
|
|
|
|
1996-03-02 18:24:13 +00:00
|
|
|
#include "opt_param.h"
|
2011-01-21 10:26:26 +00:00
|
|
|
#include "opt_msgbuf.h"
|
Make MAXPHYS tunable. Bump MAXPHYS to 1M.
Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.
Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.
Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.
Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.
Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225
2020-11-28 12:12:51 +00:00
|
|
|
#include "opt_maxphys.h"
|
2001-07-26 23:04:03 +00:00
|
|
|
#include "opt_maxusers.h"
|
1996-01-04 20:29:06 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
#include <sys/param.h>
|
2001-07-26 23:04:03 +00:00
|
|
|
#include <sys/systm.h>
|
Allocate pager bufs from UMA instead of 80-ish mutex protected linked list.
o In vm_pager_bufferinit() create pbuf_zone and start accounting on how many
pbufs are we going to have set.
In various subsystems that are going to utilize pbufs create private zones
via call to pbuf_zsecond_create(). The latter calls uma_zsecond_create(),
and sets a limit on created zone. After startup preallocate pbufs according
to requirements of all pbuf zones.
Subsystems that used to have a private limit with old allocator now have
private pbuf zones: md(4), fusefs, NFS client, smbfs, VFS cluster, FFS,
swap, vnode pager.
The following subsystems use shared pbuf zone: cam(4), nvme(4), physio(9),
aio(4). They should have their private limits, but changing that is out of
scope of this commit.
o Fetch tunable value of kern.nswbuf from init_param2() and while here move
NSWBUF_MIN to opt_param.h and eliminate opt_swap.h, that was holding only
this option.
Default values aren't touched by this commit, but they probably should be
reviewed wrt to modern hardware.
This change removes a tight bottleneck from sendfile(2) operation, that
uses pbufs in vnode pager. Other pagers also would benefit from faster
allocation.
Together with: gallatin
Tested by: pho
2019-01-15 01:02:16 +00:00
|
|
|
#include <sys/buf.h>
|
2001-07-26 23:04:03 +00:00
|
|
|
#include <sys/kernel.h>
|
2012-08-15 15:56:21 +00:00
|
|
|
#include <sys/limits.h>
|
2011-01-21 10:26:26 +00:00
|
|
|
#include <sys/msgbuf.h>
|
2012-08-15 15:56:21 +00:00
|
|
|
#include <sys/sysctl.h>
|
|
|
|
|
#include <sys/proc.h>
|
2020-11-26 18:00:26 +00:00
|
|
|
#include <sys/vnode.h>
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2011-03-23 16:38:29 +00:00
|
|
|
#include <vm/vm.h>
|
2004-11-08 18:20:02 +00:00
|
|
|
#include <vm/vm_param.h>
|
2011-03-23 16:38:29 +00:00
|
|
|
#include <vm/pmap.h>
|
2001-10-10 23:06:54 +00:00
|
|
|
|
1994-05-24 10:09:53 +00:00
|
|
|
/*
|
|
|
|
|
* System parameter formulae.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef HZ
|
2021-06-17 02:00:14 +00:00
|
|
|
# define HZ 1000
|
2008-10-27 06:25:02 +00:00
|
|
|
# ifndef HZ_VM
|
2009-07-08 01:09:12 +00:00
|
|
|
# define HZ_VM 100
|
2008-10-27 06:25:02 +00:00
|
|
|
# endif
|
|
|
|
|
#else
|
|
|
|
|
# ifndef HZ_VM
|
|
|
|
|
# define HZ_VM HZ
|
|
|
|
|
# endif
|
2004-03-14 05:49:31 +00:00
|
|
|
#endif
|
2001-07-26 23:04:03 +00:00
|
|
|
#define NPROC (20 + 16 * maxusers)
|
|
|
|
|
#ifndef NBUF
|
|
|
|
|
#define NBUF 0
|
|
|
|
|
#endif
|
1999-04-09 16:28:11 +00:00
|
|
|
#ifndef MAXFILES
|
2015-09-21 15:02:59 +00:00
|
|
|
#define MAXFILES (40 + 32 * maxusers)
|
1999-04-09 16:28:11 +00:00
|
|
|
#endif
|
1998-11-05 14:28:26 +00:00
|
|
|
|
2008-12-18 15:34:38 +00:00
|
|
|
static int sysctl_kern_vm_guest(SYSCTL_HANDLER_ARGS);
|
|
|
|
|
|
- Make callout(9) tickless, relying on eventtimers(4) as backend for
precise time event generation. This greatly improves granularity of
callouts which are not anymore constrained to wait next tick to be
scheduled.
- Extend the callout KPI introducing a set of callout_reset_sbt* functions,
which take a sbintime_t as timeout argument. The new KPI also offers a
way for consumers to specify precision tolerance they allow, so that
callout can coalesce events and reduce number of interrupts as well as
potentially avoid scheduling a SWI thread.
- Introduce support for dispatching callouts directly from hardware
interrupt context, specifying an additional flag. This feature should be
used carefully, as long as interrupt context has some limitations
(e.g. no sleeping locks can be held).
- Enhance mechanisms to gather informations about callwheel, introducing
a new sysctl to obtain stats.
This change breaks the KBI. struct callout fields has been changed, in
particular 'int ticks' (4 bytes) has been replaced with 'sbintime_t'
(8 bytes) and another 'sbintime_t' field was added for precision.
Together with: mav
Reviewed by: attilio, bde, luigi, phk
Sponsored by: Google Summer of Code 2012, iXsystems inc.
Tested by: flo (amd64, sparc64), marius (sparc64), ian (arm),
markj (amd64), mav, Fabian Keil
2013-03-04 11:09:56 +00:00
|
|
|
int hz; /* system clock's frequency */
|
|
|
|
|
int tick; /* usec per tick (1000000 / hz) */
|
2022-10-20 16:49:51 +00:00
|
|
|
time_t tick_seconds_max; /* max hz * seconds an integer can hold */
|
- Make callout(9) tickless, relying on eventtimers(4) as backend for
precise time event generation. This greatly improves granularity of
callouts which are not anymore constrained to wait next tick to be
scheduled.
- Extend the callout KPI introducing a set of callout_reset_sbt* functions,
which take a sbintime_t as timeout argument. The new KPI also offers a
way for consumers to specify precision tolerance they allow, so that
callout can coalesce events and reduce number of interrupts as well as
potentially avoid scheduling a SWI thread.
- Introduce support for dispatching callouts directly from hardware
interrupt context, specifying an additional flag. This feature should be
used carefully, as long as interrupt context has some limitations
(e.g. no sleeping locks can be held).
- Enhance mechanisms to gather informations about callwheel, introducing
a new sysctl to obtain stats.
This change breaks the KBI. struct callout fields has been changed, in
particular 'int ticks' (4 bytes) has been replaced with 'sbintime_t'
(8 bytes) and another 'sbintime_t' field was added for precision.
Together with: mav
Reviewed by: attilio, bde, luigi, phk
Sponsored by: Google Summer of Code 2012, iXsystems inc.
Tested by: flo (amd64, sparc64), marius (sparc64), ian (arm),
markj (amd64), mav, Fabian Keil
2013-03-04 11:09:56 +00:00
|
|
|
struct bintime tick_bt; /* bintime per tick (1s / hz) */
|
|
|
|
|
sbintime_t tick_sbt;
|
2001-07-26 23:04:03 +00:00
|
|
|
int maxusers; /* base tunable */
|
|
|
|
|
int maxproc; /* maximum # of processes */
|
|
|
|
|
int maxprocperuid; /* max # of procs per user */
|
|
|
|
|
int maxfiles; /* sys. wide open files limit */
|
|
|
|
|
int maxfilesperproc; /* per-proc open files limit */
|
2011-01-21 10:26:26 +00:00
|
|
|
int msgbufsize; /* size of kernel message buffer */
|
Make MAXPHYS tunable. Bump MAXPHYS to 1M.
Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.
Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.
Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.
Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.
Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225
2020-11-28 12:12:51 +00:00
|
|
|
int nbuf; /* number of bcache bufs */
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
int bio_transient_maxcnt;
|
2010-01-12 07:49:34 +00:00
|
|
|
int ngroups_max; /* max # groups per process */
|
2001-07-26 23:04:03 +00:00
|
|
|
int nswbuf;
|
2012-08-15 15:56:21 +00:00
|
|
|
pid_t pid_max = PID_MAX;
|
Make MAXPHYS tunable. Bump MAXPHYS to 1M.
Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.
Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.
Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.
Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.
Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225
2020-11-28 12:12:51 +00:00
|
|
|
u_long maxswzone; /* max swmeta KVA storage */
|
|
|
|
|
u_long maxbcache; /* max buffer cache KVA storage */
|
|
|
|
|
u_long maxpipekva; /* Limit on pipe KVA */
|
|
|
|
|
u_long maxphys; /* max raw I/O transfer size */
|
2014-10-28 19:17:44 +00:00
|
|
|
int vm_guest = VM_GUEST_NO; /* Running as virtual machine guest? */
|
2004-11-08 18:20:02 +00:00
|
|
|
u_long maxtsiz; /* max text size */
|
|
|
|
|
u_long dfldsiz; /* initial data size limit */
|
|
|
|
|
u_long maxdsiz; /* max data size */
|
|
|
|
|
u_long dflssiz; /* initial stack size limit */
|
|
|
|
|
u_long maxssiz; /* max stack size */
|
|
|
|
|
u_long sgrowsiz; /* amount to grow stack */
|
1994-05-24 10:09:53 +00:00
|
|
|
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_INT(_kern, OID_AUTO, hz, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &hz, 0,
|
2009-03-23 20:18:06 +00:00
|
|
|
"Number of clock ticks per second");
|
2022-10-03 09:08:53 +00:00
|
|
|
SYSCTL_INT(_kern, OID_AUTO, hz_max, CTLFLAG_RD, SYSCTL_NULL_INT_PTR, HZ_MAXIMUM,
|
|
|
|
|
"Maximum hz value supported");
|
|
|
|
|
SYSCTL_INT(_kern, OID_AUTO, hz_min, CTLFLAG_RD, SYSCTL_NULL_INT_PTR, HZ_MINIMUM,
|
|
|
|
|
"Minimum hz value supported");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_INT(_kern, OID_AUTO, nbuf, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &nbuf, 0,
|
2009-03-23 20:18:06 +00:00
|
|
|
"Number of buffers in the buffer cache");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_INT(_kern, OID_AUTO, nswbuf, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &nswbuf, 0,
|
2009-03-12 17:21:58 +00:00
|
|
|
"Number of swap buffers");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_INT(_kern, OID_AUTO, msgbufsize, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &msgbufsize, 0,
|
2011-01-21 10:26:26 +00:00
|
|
|
"Size of the kernel message buffer");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_LONG(_kern, OID_AUTO, maxswzone, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &maxswzone, 0,
|
2009-03-23 20:18:06 +00:00
|
|
|
"Maximum memory for swap metadata");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_LONG(_kern, OID_AUTO, maxbcache, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &maxbcache, 0,
|
2009-03-23 20:18:06 +00:00
|
|
|
"Maximum value of vfs.maxbufspace");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_INT(_kern, OID_AUTO, bio_transient_maxcnt, CTLFLAG_RDTUN | CTLFLAG_NOFETCH,
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
&bio_transient_maxcnt, 0,
|
|
|
|
|
"Maximum number of transient BIOs mappings");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_ULONG(_kern, OID_AUTO, maxtsiz, CTLFLAG_RWTUN | CTLFLAG_NOFETCH, &maxtsiz, 0,
|
2009-03-23 20:18:06 +00:00
|
|
|
"Maximum text size");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_ULONG(_kern, OID_AUTO, dfldsiz, CTLFLAG_RWTUN | CTLFLAG_NOFETCH, &dfldsiz, 0,
|
2009-03-23 20:18:06 +00:00
|
|
|
"Initial data size limit");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_ULONG(_kern, OID_AUTO, maxdsiz, CTLFLAG_RWTUN | CTLFLAG_NOFETCH, &maxdsiz, 0,
|
2009-03-23 20:18:06 +00:00
|
|
|
"Maximum data size");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_ULONG(_kern, OID_AUTO, dflssiz, CTLFLAG_RWTUN | CTLFLAG_NOFETCH, &dflssiz, 0,
|
2009-03-23 20:18:06 +00:00
|
|
|
"Initial stack size limit");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_ULONG(_kern, OID_AUTO, maxssiz, CTLFLAG_RWTUN | CTLFLAG_NOFETCH, &maxssiz, 0,
|
2009-03-23 20:18:06 +00:00
|
|
|
"Maximum stack size");
|
2014-06-28 03:56:17 +00:00
|
|
|
SYSCTL_ULONG(_kern, OID_AUTO, sgrowsiz, CTLFLAG_RWTUN | CTLFLAG_NOFETCH, &sgrowsiz, 0,
|
2012-09-03 09:26:56 +00:00
|
|
|
"Amount to grow stack on a stack fault");
|
2020-02-26 14:26:36 +00:00
|
|
|
SYSCTL_PROC(_kern, OID_AUTO, vm_guest,
|
|
|
|
|
CTLFLAG_RD | CTLTYPE_STRING | CTLFLAG_MPSAFE, NULL, 0,
|
|
|
|
|
sysctl_kern_vm_guest, "A",
|
2014-10-28 19:17:44 +00:00
|
|
|
"Virtual machine guest detected?");
|
2007-10-16 10:40:53 +00:00
|
|
|
|
2010-02-27 18:00:57 +00:00
|
|
|
/*
|
|
|
|
|
* The elements of this array are ordered based upon the values of the
|
|
|
|
|
* corresponding enum VM_GUEST members.
|
|
|
|
|
*/
|
2008-12-27 17:19:16 +00:00
|
|
|
static const char *const vm_guest_sysctl_names[] = {
|
2019-05-21 13:29:53 +00:00
|
|
|
[VM_GUEST_NO] = "none",
|
|
|
|
|
[VM_GUEST_VM] = "generic",
|
|
|
|
|
[VM_GUEST_XEN] = "xen",
|
|
|
|
|
[VM_GUEST_HV] = "hv",
|
|
|
|
|
[VM_GUEST_VMWARE] = "vmware",
|
|
|
|
|
[VM_GUEST_KVM] = "kvm",
|
|
|
|
|
[VM_GUEST_BHYVE] = "bhyve",
|
|
|
|
|
[VM_GUEST_VBOX] = "vbox",
|
|
|
|
|
[VM_GUEST_PARALLELS] = "parallels",
|
|
|
|
|
[VM_LAST] = NULL
|
2008-12-27 17:19:16 +00:00
|
|
|
};
|
2013-11-12 20:13:10 +00:00
|
|
|
CTASSERT(nitems(vm_guest_sysctl_names) - 1 == VM_LAST);
|
2008-12-27 17:19:16 +00:00
|
|
|
|
2001-07-26 23:04:03 +00:00
|
|
|
/*
|
2001-12-09 01:57:09 +00:00
|
|
|
* Boot time overrides that are not scaled against main memory
|
2001-07-26 23:04:03 +00:00
|
|
|
*/
|
|
|
|
|
void
|
2001-12-09 01:57:09 +00:00
|
|
|
init_param1(void)
|
2001-07-26 23:04:03 +00:00
|
|
|
{
|
2014-10-28 19:17:44 +00:00
|
|
|
|
2023-03-01 11:05:12 +00:00
|
|
|
#if !defined(__arm64__)
|
2015-08-10 17:18:21 +00:00
|
|
|
TUNABLE_INT_FETCH("kern.kstack_pages", &kstack_pages);
|
|
|
|
|
#endif
|
2008-10-27 06:25:02 +00:00
|
|
|
hz = -1;
|
2001-07-26 23:04:03 +00:00
|
|
|
TUNABLE_INT_FETCH("kern.hz", &hz);
|
2008-12-08 18:39:59 +00:00
|
|
|
if (hz == -1)
|
Introduce a sysctl kern.vm_guest that reflects what the kernel knows about
it running under a virtual environment. This also introduces a globally
accessible variable vm_guest that can be used where appropriate in the
kernel to inspect this environment.
To make it easier for the long run, an enum VM_GUEST is also introduced,
which could possibly be factored out in a header somewhere (but the
question is where - vm/vm_param.h? sys/param.h?) so it eventually becomes
a part of the standard KPI. In any case, it's a start.
The purpose of all this isn't to absolutely detect that the OS is running
under a virtual environment (cf. "redpill") but to allow the parts of the
kernel and the userland that care about this particular aspect and can do
something useful depending on it to have a standardised interface. Reducing
kern.hz is one example but there are other things that could be done like
avoiding context switches, not using CPU instructions that are known to be
slow in emulation, possibly different strategies in VM (memory) allocation,
CPU scheduling, etc.
It isn't clear if the JAILS/VIMAGE functionality should also be exposed
by this particular mechanism (probably not since they're not "full"
virtual hardware environments). Sometime in the future another sysctl and
a variable could be introduced to reflect if the kernel supports any kind
of virtual hosting (e.g. VMWare VMI, Xen dom0).
Reviewed by: silence from src-commiters@, virtualization@, kmacy@
Approved by: gnn (mentor)
Security: Obscurity doesn't help.
2008-12-17 19:57:12 +00:00
|
|
|
hz = vm_guest > VM_GUEST_NO ? HZ_VM : HZ;
|
2022-10-03 09:08:53 +00:00
|
|
|
|
|
|
|
|
/* range check the "hz" value */
|
|
|
|
|
if (__predict_false(hz < HZ_MINIMUM))
|
|
|
|
|
hz = HZ_MINIMUM;
|
|
|
|
|
else if (__predict_false(hz > HZ_MAXIMUM))
|
|
|
|
|
hz = HZ_MAXIMUM;
|
|
|
|
|
|
2001-07-26 23:04:03 +00:00
|
|
|
tick = 1000000 / hz;
|
- Make callout(9) tickless, relying on eventtimers(4) as backend for
precise time event generation. This greatly improves granularity of
callouts which are not anymore constrained to wait next tick to be
scheduled.
- Extend the callout KPI introducing a set of callout_reset_sbt* functions,
which take a sbintime_t as timeout argument. The new KPI also offers a
way for consumers to specify precision tolerance they allow, so that
callout can coalesce events and reduce number of interrupts as well as
potentially avoid scheduling a SWI thread.
- Introduce support for dispatching callouts directly from hardware
interrupt context, specifying an additional flag. This feature should be
used carefully, as long as interrupt context has some limitations
(e.g. no sleeping locks can be held).
- Enhance mechanisms to gather informations about callwheel, introducing
a new sysctl to obtain stats.
This change breaks the KBI. struct callout fields has been changed, in
particular 'int ticks' (4 bytes) has been replaced with 'sbintime_t'
(8 bytes) and another 'sbintime_t' field was added for precision.
Together with: mav
Reviewed by: attilio, bde, luigi, phk
Sponsored by: Google Summer of Code 2012, iXsystems inc.
Tested by: flo (amd64, sparc64), marius (sparc64), ian (arm),
markj (amd64), mav, Fabian Keil
2013-03-04 11:09:56 +00:00
|
|
|
tick_sbt = SBT_1S / hz;
|
|
|
|
|
tick_bt = sbttobt(tick_sbt);
|
2022-10-20 16:49:51 +00:00
|
|
|
tick_seconds_max = INT_MAX / hz;
|
2001-07-26 23:04:03 +00:00
|
|
|
|
2016-11-22 01:02:59 +00:00
|
|
|
/*
|
|
|
|
|
* Arrange for ticks to wrap 10 minutes after boot to help catch
|
|
|
|
|
* sign problems sooner.
|
|
|
|
|
*/
|
|
|
|
|
ticks = INT_MAX - (hz * 10 * 60);
|
|
|
|
|
|
2020-11-26 18:00:26 +00:00
|
|
|
vn_lock_pair_pause_max = hz / 100;
|
|
|
|
|
if (vn_lock_pair_pause_max == 0)
|
|
|
|
|
vn_lock_pair_pause_max = 1;
|
|
|
|
|
|
2001-08-20 16:29:13 +00:00
|
|
|
#ifdef VM_SWZONE_SIZE_MAX
|
2001-08-20 00:41:12 +00:00
|
|
|
maxswzone = VM_SWZONE_SIZE_MAX;
|
2001-08-20 16:29:13 +00:00
|
|
|
#endif
|
Adjust some variables (mostly related to the buffer cache) that hold
address space sizes to be longs instead of ints. Specifically, the follow
values are now longs: runningbufspace, bufspace, maxbufspace,
bufmallocspace, maxbufmallocspace, lobufspace, hibufspace, lorunningspace,
hirunningspace, maxswzone, maxbcache, and maxpipekva. Previously, a
relatively small number (~ 44000) of buffers set in kern.nbuf would result
in integer overflows resulting either in hangs or bogus values of
hidirtybuffers and lodirtybuffers. Now one has to overflow a long to see
such problems. There was a check for a nbuf setting that would cause
overflows in the auto-tuning of nbuf. I've changed it to always check and
cap nbuf but warn if a user-supplied tunable would cause overflow.
Note that this changes the ABI of several sysctls that are used by things
like top(1), etc., so any MFC would probably require a some gross shims
to allow for that.
MFC after: 1 month
2009-03-09 19:35:20 +00:00
|
|
|
TUNABLE_LONG_FETCH("kern.maxswzone", &maxswzone);
|
2001-08-20 16:29:13 +00:00
|
|
|
#ifdef VM_BCACHE_SIZE_MAX
|
2001-08-20 00:41:12 +00:00
|
|
|
maxbcache = VM_BCACHE_SIZE_MAX;
|
2001-08-20 16:29:13 +00:00
|
|
|
#endif
|
Adjust some variables (mostly related to the buffer cache) that hold
address space sizes to be longs instead of ints. Specifically, the follow
values are now longs: runningbufspace, bufspace, maxbufspace,
bufmallocspace, maxbufmallocspace, lobufspace, hibufspace, lorunningspace,
hirunningspace, maxswzone, maxbcache, and maxpipekva. Previously, a
relatively small number (~ 44000) of buffers set in kern.nbuf would result
in integer overflows resulting either in hangs or bogus values of
hidirtybuffers and lodirtybuffers. Now one has to overflow a long to see
such problems. There was a check for a nbuf setting that would cause
overflows in the auto-tuning of nbuf. I've changed it to always check and
cap nbuf but warn if a user-supplied tunable would cause overflow.
Note that this changes the ABI of several sysctls that are used by things
like top(1), etc., so any MFC would probably require a some gross shims
to allow for that.
MFC after: 1 month
2009-03-09 19:35:20 +00:00
|
|
|
TUNABLE_LONG_FETCH("kern.maxbcache", &maxbcache);
|
2011-01-21 10:26:26 +00:00
|
|
|
msgbufsize = MSGBUF_SIZE;
|
|
|
|
|
TUNABLE_INT_FETCH("kern.msgbufsize", &msgbufsize);
|
2001-10-10 23:06:54 +00:00
|
|
|
|
|
|
|
|
maxtsiz = MAXTSIZ;
|
2004-11-08 18:20:02 +00:00
|
|
|
TUNABLE_ULONG_FETCH("kern.maxtsiz", &maxtsiz);
|
2001-10-10 23:06:54 +00:00
|
|
|
dfldsiz = DFLDSIZ;
|
2004-11-08 18:20:02 +00:00
|
|
|
TUNABLE_ULONG_FETCH("kern.dfldsiz", &dfldsiz);
|
2001-10-10 23:06:54 +00:00
|
|
|
maxdsiz = MAXDSIZ;
|
2004-11-08 18:20:02 +00:00
|
|
|
TUNABLE_ULONG_FETCH("kern.maxdsiz", &maxdsiz);
|
2001-10-10 23:06:54 +00:00
|
|
|
dflssiz = DFLSSIZ;
|
2004-11-08 18:20:02 +00:00
|
|
|
TUNABLE_ULONG_FETCH("kern.dflssiz", &dflssiz);
|
2001-10-10 23:06:54 +00:00
|
|
|
maxssiz = MAXSSIZ;
|
2004-11-08 18:20:02 +00:00
|
|
|
TUNABLE_ULONG_FETCH("kern.maxssiz", &maxssiz);
|
2001-10-10 23:06:54 +00:00
|
|
|
sgrowsiz = SGROWSIZ;
|
2004-11-08 18:20:02 +00:00
|
|
|
TUNABLE_ULONG_FETCH("kern.sgrowsiz", &sgrowsiz);
|
2010-01-12 07:49:34 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Let the administrator set {NGROUPS_MAX}, but disallow values
|
|
|
|
|
* less than NGROUPS_MAX which would violate POSIX.1-2008 or
|
|
|
|
|
* greater than INT_MAX-1 which would result in overflow.
|
|
|
|
|
*/
|
|
|
|
|
ngroups_max = NGROUPS_MAX;
|
|
|
|
|
TUNABLE_INT_FETCH("kern.ngroups", &ngroups_max);
|
|
|
|
|
if (ngroups_max < NGROUPS_MAX)
|
|
|
|
|
ngroups_max = NGROUPS_MAX;
|
2012-08-15 15:56:21 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Only allow to lower the maximal pid.
|
2012-08-16 13:04:21 +00:00
|
|
|
* Prevent setting up a non-bootable system if pid_max is too low.
|
2012-08-15 15:56:21 +00:00
|
|
|
*/
|
|
|
|
|
TUNABLE_INT_FETCH("kern.pid_max", &pid_max);
|
|
|
|
|
if (pid_max > PID_MAX)
|
|
|
|
|
pid_max = PID_MAX;
|
2012-08-16 13:04:21 +00:00
|
|
|
else if (pid_max < 300)
|
|
|
|
|
pid_max = 300;
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
|
|
|
|
|
TUNABLE_INT_FETCH("vfs.unmapped_buf_allowed", &unmapped_buf_allowed);
|
2001-07-26 23:04:03 +00:00
|
|
|
}
|
2001-12-09 01:57:09 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Boot time overrides that are scaled against main memory
|
|
|
|
|
*/
|
|
|
|
|
void
|
2002-08-30 04:04:37 +00:00
|
|
|
init_param2(long physpages)
|
2001-12-09 01:57:09 +00:00
|
|
|
{
|
|
|
|
|
|
|
|
|
|
/* Base parameters */
|
2002-02-06 01:19:19 +00:00
|
|
|
maxusers = MAXUSERS;
|
|
|
|
|
TUNABLE_INT_FETCH("kern.maxusers", &maxusers);
|
|
|
|
|
if (maxusers == 0) {
|
2002-01-25 01:54:16 +00:00
|
|
|
maxusers = physpages / (2 * 1024 * 1024 / PAGE_SIZE);
|
2001-12-09 01:57:09 +00:00
|
|
|
if (maxusers < 32)
|
|
|
|
|
maxusers = 32;
|
2012-11-10 02:08:40 +00:00
|
|
|
#ifdef VM_MAX_AUTOTUNE_MAXUSERS
|
|
|
|
|
if (maxusers > VM_MAX_AUTOTUNE_MAXUSERS)
|
|
|
|
|
maxusers = VM_MAX_AUTOTUNE_MAXUSERS;
|
|
|
|
|
#endif
|
|
|
|
|
/*
|
|
|
|
|
* Scales down the function in which maxusers grows once
|
|
|
|
|
* we hit 384.
|
|
|
|
|
*/
|
|
|
|
|
if (maxusers > 384)
|
|
|
|
|
maxusers = 384 + ((maxusers - 384) / 8);
|
|
|
|
|
}
|
2001-12-09 01:57:09 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* The following can be overridden after boot via sysctl. Note:
|
2022-09-03 13:26:55 +00:00
|
|
|
* unless overridden, these macros are ultimately based on maxusers.
|
2002-03-07 04:50:36 +00:00
|
|
|
* Limit maxproc so that kmap entries cannot be exhausted by
|
|
|
|
|
* processes.
|
|
|
|
|
*/
|
Base the mbuf related limits on the available physical memory or
kernel memory, whichever is lower. The overall mbuf related memory
limit must be set so that mbufs (and clusters of various sizes)
can't exhaust physical RAM or KVM.
The limit is set to half of the physical RAM or KVM (whichever is
lower) as the baseline. In any normal scenario we want to leave
at least half of the physmem/kvm for other kernel functions and
userspace to prevent it from swapping too easily. Via a tunable
kern.maxmbufmem the limit can be upped to at most 3/4 of physmem/kvm.
At the same time divorce maxfiles from maxusers and set maxfiles to
physpages / 8 with a floor based on maxusers. This way busy servers
can make use of the significantly increased mbuf limits with a much
larger number of open sockets.
Tidy up ordering in init_param2() and check up on some users of
those values calculated here.
Out of the overall mbuf memory limit 2K clusters and 4K (page size)
clusters to get 1/4 each because these are the most heavily used mbuf
sizes. 2K clusters are used for MTU 1500 ethernet inbound packets.
4K clusters are used whenever possible for sends on sockets and thus
outbound packets. The larger cluster sizes of 9K and 16K are limited
to 1/6 of the overall mbuf memory limit. When jumbo MTU's are used
these large clusters will end up only on the inbound path. They are
not used on outbound, there it's still 4K. Yes, that will stay that
way because otherwise we run into lots of complications in the
stack. And it really isn't a problem, so don't make a scene.
Normal mbufs (256B) weren't limited at all previously. This was
problematic as there are certain places in the kernel that on
allocation failure of clusters try to piece together their packet
from smaller mbufs.
The mbuf limit is the number of all other mbuf sizes together plus
some more to allow for standalone mbufs (ACK for example) and to
send off a copy of a cluster. Unfortunately there isn't a way to
set an overall limit for all mbuf memory together as UMA doesn't
support such a limiting.
NB: Every cluster also has an mbuf associated with it.
Two examples on the revised mbuf sizing limits:
1GB KVM:
512MB limit for mbufs
419,430 mbufs
65,536 2K mbuf clusters
32,768 4K mbuf clusters
9,709 9K mbuf clusters
5,461 16K mbuf clusters
16GB RAM:
8GB limit for mbufs
33,554,432 mbufs
1,048,576 2K mbuf clusters
524,288 4K mbuf clusters
155,344 9K mbuf clusters
87,381 16K mbuf clusters
These defaults should be sufficient for even the most demanding
network loads.
MFC after: 1 month
2012-11-27 21:19:58 +00:00
|
|
|
maxproc = NPROC;
|
|
|
|
|
TUNABLE_INT_FETCH("kern.maxproc", &maxproc);
|
2002-03-07 04:50:36 +00:00
|
|
|
if (maxproc > (physpages / 12))
|
|
|
|
|
maxproc = physpages / 12;
|
2015-09-21 15:02:59 +00:00
|
|
|
if (maxproc > pid_max)
|
|
|
|
|
maxproc = pid_max;
|
2001-12-13 20:00:45 +00:00
|
|
|
maxprocperuid = (maxproc * 9) / 10;
|
Base the mbuf related limits on the available physical memory or
kernel memory, whichever is lower. The overall mbuf related memory
limit must be set so that mbufs (and clusters of various sizes)
can't exhaust physical RAM or KVM.
The limit is set to half of the physical RAM or KVM (whichever is
lower) as the baseline. In any normal scenario we want to leave
at least half of the physmem/kvm for other kernel functions and
userspace to prevent it from swapping too easily. Via a tunable
kern.maxmbufmem the limit can be upped to at most 3/4 of physmem/kvm.
At the same time divorce maxfiles from maxusers and set maxfiles to
physpages / 8 with a floor based on maxusers. This way busy servers
can make use of the significantly increased mbuf limits with a much
larger number of open sockets.
Tidy up ordering in init_param2() and check up on some users of
those values calculated here.
Out of the overall mbuf memory limit 2K clusters and 4K (page size)
clusters to get 1/4 each because these are the most heavily used mbuf
sizes. 2K clusters are used for MTU 1500 ethernet inbound packets.
4K clusters are used whenever possible for sends on sockets and thus
outbound packets. The larger cluster sizes of 9K and 16K are limited
to 1/6 of the overall mbuf memory limit. When jumbo MTU's are used
these large clusters will end up only on the inbound path. They are
not used on outbound, there it's still 4K. Yes, that will stay that
way because otherwise we run into lots of complications in the
stack. And it really isn't a problem, so don't make a scene.
Normal mbufs (256B) weren't limited at all previously. This was
problematic as there are certain places in the kernel that on
allocation failure of clusters try to piece together their packet
from smaller mbufs.
The mbuf limit is the number of all other mbuf sizes together plus
some more to allow for standalone mbufs (ACK for example) and to
send off a copy of a cluster. Unfortunately there isn't a way to
set an overall limit for all mbuf memory together as UMA doesn't
support such a limiting.
NB: Every cluster also has an mbuf associated with it.
Two examples on the revised mbuf sizing limits:
1GB KVM:
512MB limit for mbufs
419,430 mbufs
65,536 2K mbuf clusters
32,768 4K mbuf clusters
9,709 9K mbuf clusters
5,461 16K mbuf clusters
16GB RAM:
8GB limit for mbufs
33,554,432 mbufs
1,048,576 2K mbuf clusters
524,288 4K mbuf clusters
155,344 9K mbuf clusters
87,381 16K mbuf clusters
These defaults should be sufficient for even the most demanding
network loads.
MFC after: 1 month
2012-11-27 21:19:58 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* The default limit for maxfiles is 1/12 of the number of
|
|
|
|
|
* physical page but not less than 16 times maxusers.
|
|
|
|
|
* At most it can be 1/6 the number of physical pages.
|
|
|
|
|
*/
|
|
|
|
|
maxfiles = imax(MAXFILES, physpages / 8);
|
|
|
|
|
TUNABLE_INT_FETCH("kern.maxfiles", &maxfiles);
|
|
|
|
|
if (maxfiles > (physpages / 4))
|
|
|
|
|
maxfiles = physpages / 4;
|
|
|
|
|
maxfilesperproc = (maxfiles / 10) * 9;
|
Also make kern.maxfilesperproc a boot time tunable.
Auto-tuning threshold discussions aside, it turns out that if you want
to lower this on say, rather memory-packed machines, you either set maxusers
or kern.maxfiles, or you set it in sysctl. The former is a non-exact
way to tune this; the latter doesn't actually affect anything in the
startup scripts.
This first occured because I wondered why the hell screen would take upwards
of 10 seconds to spawn a new screen. I then found python doing the same
thing during fork/exec of child processes - it calls close() on each FD
up to the current openfiles limit. On a 1TB machine this is like, 26 million
FDs per process. Ugh.
So:
* This allows it to be set early in /boot/loader.conf;
* It can be used to work around the ridiculous situation of
screen, python, etc doing a close() on potentially millions of FDs
even though you only have four open.
Tested:
* 4GB, 32GB, 64GB, 128GB, 384GB, 1TB systems with autotune, ensuring
screen and python forking doesn't result in some pretty hilariously
bad behaviour.
TODO:
* Note that the default login.conf sets openfiles-cur to unlimited,
effectively obeying kern.maxfilesperproc. Perhaps we should fix
this.
* .. and even if we do, we need to also ensure that daemons get
a soft limit of something reasonable and capped - they can request
more FDs themselves.
MFC after: 1 week
Sponsored by: Norse Corp, Inc.
2015-09-10 04:05:58 +00:00
|
|
|
TUNABLE_INT_FETCH("kern.maxfilesperproc", &maxfilesperproc);
|
|
|
|
|
|
2003-07-11 00:01:03 +00:00
|
|
|
/*
|
|
|
|
|
* Cannot be changed after boot.
|
|
|
|
|
*/
|
|
|
|
|
nbuf = NBUF;
|
|
|
|
|
TUNABLE_INT_FETCH("kern.nbuf", &nbuf);
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
TUNABLE_INT_FETCH("kern.bio_transient_maxcnt", &bio_transient_maxcnt);
|
Make MAXPHYS tunable. Bump MAXPHYS to 1M.
Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.
Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.
Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.
Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.
Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225
2020-11-28 12:12:51 +00:00
|
|
|
maxphys = MAXPHYS;
|
|
|
|
|
TUNABLE_ULONG_FETCH("kern.maxphys", &maxphys);
|
|
|
|
|
if (maxphys == 0) {
|
|
|
|
|
maxphys = MAXPHYS;
|
|
|
|
|
} else if (__bitcountl(maxphys) != 1) { /* power of two */
|
|
|
|
|
if (flsl(maxphys) == NBBY * sizeof(maxphys))
|
|
|
|
|
maxphys = MAXPHYS;
|
|
|
|
|
else
|
|
|
|
|
maxphys = 1UL << flsl(maxphys);
|
|
|
|
|
}
|
|
|
|
|
if (maxphys < PAGE_SIZE)
|
|
|
|
|
maxphys = MAXPHYS;
|
2003-07-11 00:01:03 +00:00
|
|
|
|
Allocate pager bufs from UMA instead of 80-ish mutex protected linked list.
o In vm_pager_bufferinit() create pbuf_zone and start accounting on how many
pbufs are we going to have set.
In various subsystems that are going to utilize pbufs create private zones
via call to pbuf_zsecond_create(). The latter calls uma_zsecond_create(),
and sets a limit on created zone. After startup preallocate pbufs according
to requirements of all pbuf zones.
Subsystems that used to have a private limit with old allocator now have
private pbuf zones: md(4), fusefs, NFS client, smbfs, VFS cluster, FFS,
swap, vnode pager.
The following subsystems use shared pbuf zone: cam(4), nvme(4), physio(9),
aio(4). They should have their private limits, but changing that is out of
scope of this commit.
o Fetch tunable value of kern.nswbuf from init_param2() and while here move
NSWBUF_MIN to opt_param.h and eliminate opt_swap.h, that was holding only
this option.
Default values aren't touched by this commit, but they probably should be
reviewed wrt to modern hardware.
This change removes a tight bottleneck from sendfile(2) operation, that
uses pbufs in vnode pager. Other pagers also would benefit from faster
allocation.
Together with: gallatin
Tested by: pho
2019-01-15 01:02:16 +00:00
|
|
|
/*
|
|
|
|
|
* Physical buffers are pre-allocated buffers (struct buf) that
|
|
|
|
|
* are used as temporary holders for I/O, such as paging I/O.
|
|
|
|
|
*/
|
|
|
|
|
TUNABLE_INT_FETCH("kern.nswbuf", &nswbuf);
|
|
|
|
|
|
2003-07-08 04:02:31 +00:00
|
|
|
/*
|
2011-03-23 16:38:29 +00:00
|
|
|
* The default for maxpipekva is min(1/64 of the kernel address space,
|
|
|
|
|
* max(1/64 of main memory, 512KB)). See sys_pipe.c for more details.
|
2003-07-08 04:02:31 +00:00
|
|
|
*/
|
Make MAXPHYS tunable. Bump MAXPHYS to 1M.
Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.
Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.
Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.
Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.
Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225
2020-11-28 12:12:51 +00:00
|
|
|
maxpipekva = ptoa(physpages / 64);
|
Base the mbuf related limits on the available physical memory or
kernel memory, whichever is lower. The overall mbuf related memory
limit must be set so that mbufs (and clusters of various sizes)
can't exhaust physical RAM or KVM.
The limit is set to half of the physical RAM or KVM (whichever is
lower) as the baseline. In any normal scenario we want to leave
at least half of the physmem/kvm for other kernel functions and
userspace to prevent it from swapping too easily. Via a tunable
kern.maxmbufmem the limit can be upped to at most 3/4 of physmem/kvm.
At the same time divorce maxfiles from maxusers and set maxfiles to
physpages / 8 with a floor based on maxusers. This way busy servers
can make use of the significantly increased mbuf limits with a much
larger number of open sockets.
Tidy up ordering in init_param2() and check up on some users of
those values calculated here.
Out of the overall mbuf memory limit 2K clusters and 4K (page size)
clusters to get 1/4 each because these are the most heavily used mbuf
sizes. 2K clusters are used for MTU 1500 ethernet inbound packets.
4K clusters are used whenever possible for sends on sockets and thus
outbound packets. The larger cluster sizes of 9K and 16K are limited
to 1/6 of the overall mbuf memory limit. When jumbo MTU's are used
these large clusters will end up only on the inbound path. They are
not used on outbound, there it's still 4K. Yes, that will stay that
way because otherwise we run into lots of complications in the
stack. And it really isn't a problem, so don't make a scene.
Normal mbufs (256B) weren't limited at all previously. This was
problematic as there are certain places in the kernel that on
allocation failure of clusters try to piece together their packet
from smaller mbufs.
The mbuf limit is the number of all other mbuf sizes together plus
some more to allow for standalone mbufs (ACK for example) and to
send off a copy of a cluster. Unfortunately there isn't a way to
set an overall limit for all mbuf memory together as UMA doesn't
support such a limiting.
NB: Every cluster also has an mbuf associated with it.
Two examples on the revised mbuf sizing limits:
1GB KVM:
512MB limit for mbufs
419,430 mbufs
65,536 2K mbuf clusters
32,768 4K mbuf clusters
9,709 9K mbuf clusters
5,461 16K mbuf clusters
16GB RAM:
8GB limit for mbufs
33,554,432 mbufs
1,048,576 2K mbuf clusters
524,288 4K mbuf clusters
155,344 9K mbuf clusters
87,381 16K mbuf clusters
These defaults should be sufficient for even the most demanding
network loads.
MFC after: 1 month
2012-11-27 21:19:58 +00:00
|
|
|
TUNABLE_LONG_FETCH("kern.ipc.maxpipekva", &maxpipekva);
|
2003-07-08 04:02:31 +00:00
|
|
|
if (maxpipekva < 512 * 1024)
|
|
|
|
|
maxpipekva = 512 * 1024;
|
2011-03-23 16:38:29 +00:00
|
|
|
if (maxpipekva > (VM_MAX_KERNEL_ADDRESS - VM_MIN_KERNEL_ADDRESS) / 64)
|
|
|
|
|
maxpipekva = (VM_MAX_KERNEL_ADDRESS - VM_MIN_KERNEL_ADDRESS) /
|
|
|
|
|
64;
|
2001-12-09 01:57:09 +00:00
|
|
|
}
|
2008-12-18 15:34:38 +00:00
|
|
|
|
|
|
|
|
/*
|
2013-10-27 18:52:09 +00:00
|
|
|
* Sysctl stringifying handler for kern.vm_guest.
|
2008-12-18 15:34:38 +00:00
|
|
|
*/
|
|
|
|
|
static int
|
|
|
|
|
sysctl_kern_vm_guest(SYSCTL_HANDLER_ARGS)
|
|
|
|
|
{
|
2015-03-14 21:40:01 +00:00
|
|
|
return (SYSCTL_OUT_STR(req, vm_guest_sysctl_names[vm_guest]));
|
2008-12-18 15:34:38 +00:00
|
|
|
}
|
