d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
bd7d64f548
freebsd-nq/sys/kern/uipc_mbuf.c

2137 lines
52 KiB
C
Raw Normal View History

/* -> /*- for copyright notices, minor format tweaks as necessary
2005-01-06 23:35:40 +00:00
/*-
sys: further adoption of SPDX licensing ID tags. Mainly focus on files that use BSD 3-Clause license. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts. Special thanks to Wind River for providing access to "The Duke of Highlander" tool: an older (2014) run over FreeBSD tree was useful as a starting point.
2017-11-20 19:43:44 +00:00
* SPDX-License-Identifier: BSD-3-Clause
*
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
* Copyright (c) 1982, 1986, 1988, 1991, 1993
* The Regents of the University of California. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
Renumber license clauses in sys/kern to avoid skipping #3
2016-09-15 13:16:20 +00:00
* 3. Neither the name of the University nor the names of its contributors
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* @(#)uipc_mbuf.c 8.2 (Berkeley) 1/4/94
*/
Use __FBSDID().
2003-06-11 00:56:59 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
Back out jesper's 2001/05/31 14:58:11 PDT commit. It does not compile.
2001-06-01 09:51:14 +00:00
#include "opt_param.h"
Add another MBUF_STRESS_TEST feature, m_defragrandomfailures. When enabled, this causes m_defrag to randomly return NULL (following its normal failure case so that extra memory leaks are not introduced.) Code similar to this was used to find / fix a few bugs last week.
2003-04-15 02:14:43 +00:00
#include "opt_mbuf_stress_test.h"
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
#include "opt_mbuf_profiling.h"
Introduce support for Mandatory Access Control and extensible kernel access control. Invoke additional MAC entry points when an mbuf packet header is copied to another mbuf: release the old label if any, reinitialize the new header, and ask the MAC framework to copy the header label data. Note that this requires a potential allocation operation, but m_copy_pkthdr() is not permitted to fail, so we must block. Since we now use interrupt threads, this is possible, but not desirable. Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
2002-07-31 01:51:34 +00:00
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/param.h>
#include <sys/systm.h>
Undo part of the tangle of having sys/lock.h and sys/mutex.h included in other "system" header files. Also help the deprecation of lockmgr.h by making it a sub-include of sys/lock.h and removing sys/lockmgr.h form kernel .c files. Sort sys/*.h includes where possible in affected files. OK'ed by: bde (with reservations)
2001-05-01 08:13:21 +00:00
#include <sys/kernel.h>
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
#include <sys/limits.h>
Undo part of the tangle of having sys/lock.h and sys/mutex.h included in other "system" header files. Also help the deprecation of lockmgr.h by making it a sub-include of sys/lock.h and removing sys/lockmgr.h form kernel .c files. Sort sys/*.h includes where possible in affected files. OK'ed by: bde (with reservations)
2001-05-01 08:13:21 +00:00
#include <sys/lock.h>
Include file cleanup; mac.h and malloc.h at one point had ordering relationship requirements, and no longer do. Reminded by: bde
2002-08-01 17:47:56 +00:00
#include <sys/malloc.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/mbuf.h>
Create a new branch of the kernel MIB, kern.ipc, to store all of the configurables and instrumentation related to inter-process communication mechanisms. Some variables, like mbuf statistics, are instrumented here for the first time. For mbuf statistics: also keep track of m_copym() and m_pullup() failures, and provide for the user's inspection the compiled-in values of MSIZE, MHLEN, MCLBYTES, and MINCLSIZE.
1997-02-24 20:30:58 +00:00
#include <sys/sysctl.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/domain.h>
#include <sys/protosw.h>
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
#include <sys/uio.h>
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
#include <sys/vmmeter.h>
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
#include <sys/sdt.h>
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
#include <vm/vm.h>
#include <vm/vm_pageout.h>
#include <vm/vm_page.h>
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
Move mbuf provider under SDT to indicate that it is FreeBSD specific and not a stable interface. Reviewed by: markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5716
2016-03-24 08:26:06 +00:00
SDT_PROBE_DEFINE5_XLATE(sdt, , , m__init,
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
"struct mbuf *", "mbufinfo_t *",
"uint32_t", "uint32_t",
"uint16_t", "uint16_t",
"uint32_t", "uint32_t",
"uint32_t", "uint32_t");
Move mbuf provider under SDT to indicate that it is FreeBSD specific and not a stable interface. Reviewed by: markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5716
2016-03-24 08:26:06 +00:00
SDT_PROBE_DEFINE3_XLATE(sdt, , , m__gethdr,
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
"uint32_t", "uint32_t",
"uint16_t", "uint16_t",
"struct mbuf *", "mbufinfo_t *");
Move mbuf provider under SDT to indicate that it is FreeBSD specific and not a stable interface. Reviewed by: markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5716
2016-03-24 08:26:06 +00:00
SDT_PROBE_DEFINE3_XLATE(sdt, , , m__get,
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
"uint32_t", "uint32_t",
"uint16_t", "uint16_t",
"struct mbuf *", "mbufinfo_t *");
Move mbuf provider under SDT to indicate that it is FreeBSD specific and not a stable interface. Reviewed by: markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5716
2016-03-24 08:26:06 +00:00
SDT_PROBE_DEFINE4_XLATE(sdt, , , m__getcl,
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
"uint32_t", "uint32_t",
"uint16_t", "uint16_t",
"uint32_t", "uint32_t",
"struct mbuf *", "mbufinfo_t *");
Move mbuf provider under SDT to indicate that it is FreeBSD specific and not a stable interface. Reviewed by: markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5716
2016-03-24 08:26:06 +00:00
SDT_PROBE_DEFINE3_XLATE(sdt, , , m__clget,
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
"struct mbuf *", "mbufinfo_t *",
"uint32_t", "uint32_t",
"uint32_t", "uint32_t");
Move mbuf provider under SDT to indicate that it is FreeBSD specific and not a stable interface. Reviewed by: markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5716
2016-03-24 08:26:06 +00:00
SDT_PROBE_DEFINE4_XLATE(sdt, , , m__cljget,
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
"struct mbuf *", "mbufinfo_t *",
"uint32_t", "uint32_t",
"uint32_t", "uint32_t",
"void*", "void*");
Move mbuf provider under SDT to indicate that it is FreeBSD specific and not a stable interface. Reviewed by: markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5716
2016-03-24 08:26:06 +00:00
SDT_PROBE_DEFINE(sdt, , , m__cljset);
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
Move mbuf provider under SDT to indicate that it is FreeBSD specific and not a stable interface. Reviewed by: markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5716
2016-03-24 08:26:06 +00:00
SDT_PROBE_DEFINE1_XLATE(sdt, , , m__free,
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
"struct mbuf *", "mbufinfo_t *");
Move mbuf provider under SDT to indicate that it is FreeBSD specific and not a stable interface. Reviewed by: markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5716
2016-03-24 08:26:06 +00:00
SDT_PROBE_DEFINE1_XLATE(sdt, , , m__freem,
Add an mbuf provider to DTrace. The mbuf provider is made up of a set of Statically Defined Tracepoints which help us look into mbufs as they are allocated and freed. This can be used to inspect the buffers or for a simplified mbuf leak detector. New tracepoints are: mbuf:::m-init mbuf:::m-gethdr mbuf:::m-get mbuf:::m-getcl mbuf:::m-clget mbuf:::m-cljget mbuf:::m-cljset mbuf:::m-free mbuf:::m-freem There is also a translator for mbufs which gives some visibility into the structure, see mbuf.d for more details. Reviewed by: bz, markj MFC after: 2 weeks Sponsored by: Rubicon Communications (Netgate) Differential Revision: https://reviews.freebsd.org/D5682
2016-03-22 13:16:52 +00:00
"struct mbuf *", "mbufinfo_t *");
Undo part of the tangle of having sys/lock.h and sys/mutex.h included in other "system" header files. Also help the deprecation of lockmgr.h by making it a sub-include of sys/lock.h and removing sys/lockmgr.h form kernel .c files. Sort sys/*.h includes where possible in affected files. OK'ed by: bde (with reservations)
2001-05-01 08:13:21 +00:00
o Gather all mbuf(9) allocation functions into kern_mbuf.c, and all mbuf(9) manipulation functions into uipc_mbuf.c. This looks like the initial intent, but had diffused in the last decade. o Gather all declarations in mbuf.h in one place and sort them. o Uninline m_clget() and m_cljget(). There are no functional changes in this patch. The patch comes from a larger version, where all mbuf(9) allocation was uninlined, which allowed to make mbuf(9) UMA zones private to kern_mbuf.c. The performance impact of the total uninlining is still unclear, so we are holding on now with larger version. Together with: melifaro, olivier
2016-02-11 21:32:23 +00:00
#include <security/mac/mac_framework.h>
Eliminate sloppy common-style declarations. There should be none left for the LINT configuation.
1995-07-29 11:44:31 +00:00
int max_linkhdr;
int max_protohdr;
int max_hdr;
int max_datalen;
Hide the m_defrag* statistics under MBUF_STRESS_TEST, there seems to be no need to see them in the general case (and they aren't smp-safe anyway.) Suggested by: hmp MFC after: 1 week
2003-06-17 02:34:40 +00:00
#ifdef MBUF_STRESS_TEST
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
int m_defragpackets;
int m_defragbytes;
int m_defraguseless;
int m_defragfailure;
Add another MBUF_STRESS_TEST feature, m_defragrandomfailures. When enabled, this causes m_defrag to randomly return NULL (following its normal failure case so that extra memory leaks are not introduced.) Code similar to this was used to find / fix a few bugs last week.
2003-04-15 02:14:43 +00:00
int m_defragrandomfailures;
#endif
Big mbuf subsystem diff #1: incorporate mutexes and fix things up somewhat to accomodate the changes. Here's a list of things that have changed (I may have left out a few); for a relatively complete list, see http://people.freebsd.org/~bmilekic/mtx_journal * Remove old (once useful) mcluster code for MCLBYTES > PAGE_SIZE which nobody uses anymore. It was great while it lasted, but now we're moving onto bigger and better things (Approved by: wollman). * Practically re-wrote the allocation macros in sys/sys/mbuf.h to accomodate new allocations which grab the necessary lock. * Make sure that necessary mbstat variables are manipulated with corresponding atomic() routines. * Changed the "wait" routines, cleaned it up, made one routine that does the job. * Generalized MWAKEUP() macro. Got rid of m_retry and m_retryhdr, as they are now included in the generalized "wait" routines. * Sleep routines now use msleep(). * Free lists have locks. * etc... probably other stuff I'm missing... Things to look out for and work on later: * find a better way to (dynamically) adjust EXT_COUNTERS * move necessity to recurse on a lock from drain routines by providing lock-free lower-level version of MFREE() (and possibly m_free()?). * checkout include of mutex.h in sys/sys/mbuf.h - probably violating general philosophy here. The code has been reviewed quite a bit, but problems may arise... please, don't panic! Send me Emails: bmilekic@freebsd.org Reviewed by: jlemon, cp, alfred, others?
2000-09-30 06:30:39 +00:00
/*
* sysctl(8) exported objects
*/
The sysctls kern.ipc.[max_linkhdr|max_protohdr|max_hdr|max_datalen] can't be changed from userland. Make them read-only and provide descriptions. kern.ipc.max_datalen must never be less than one byte. Enforce this with a panic in net_init_domain(). Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 days
2006-02-18 17:16:18 +00:00
SYSCTL_INT(_kern_ipc, KIPC_MAX_LINKHDR, max_linkhdr, CTLFLAG_RD,
&max_linkhdr, 0, "Size of largest link layer header");
SYSCTL_INT(_kern_ipc, KIPC_MAX_PROTOHDR, max_protohdr, CTLFLAG_RD,
&max_protohdr, 0, "Size of largest protocol layer header");
SYSCTL_INT(_kern_ipc, KIPC_MAX_HDR, max_hdr, CTLFLAG_RD,
&max_hdr, 0, "Size of largest link plus protocol header");
SYSCTL_INT(_kern_ipc, KIPC_MAX_DATALEN, max_datalen, CTLFLAG_RD,
&max_datalen, 0, "Minimum space left in mbuf after max_hdr");
Hide the m_defrag* statistics under MBUF_STRESS_TEST, there seems to be no need to see them in the general case (and they aren't smp-safe anyway.) Suggested by: hmp MFC after: 1 week
2003-06-17 02:34:40 +00:00
#ifdef MBUF_STRESS_TEST
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
SYSCTL_INT(_kern_ipc, OID_AUTO, m_defragpackets, CTLFLAG_RD,
&m_defragpackets, 0, "");
SYSCTL_INT(_kern_ipc, OID_AUTO, m_defragbytes, CTLFLAG_RD,
&m_defragbytes, 0, "");
SYSCTL_INT(_kern_ipc, OID_AUTO, m_defraguseless, CTLFLAG_RD,
&m_defraguseless, 0, "");
SYSCTL_INT(_kern_ipc, OID_AUTO, m_defragfailure, CTLFLAG_RD,
&m_defragfailure, 0, "");
Add another MBUF_STRESS_TEST feature, m_defragrandomfailures. When enabled, this causes m_defrag to randomly return NULL (following its normal failure case so that extra memory leaks are not introduced.) Code similar to this was used to find / fix a few bugs last week.
2003-04-15 02:14:43 +00:00
SYSCTL_INT(_kern_ipc, OID_AUTO, m_defragrandomfailures, CTLFLAG_RW,
&m_defragrandomfailures, 0, "");
#endif
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
Pad m_hdr on 32bit architectures to to prevent alignment and padding problems with the way MLEN, MHLEN, and struct mbuf are set up. CTASSERT's are provided to detect such issues at compile time in the future. The #define MLEN and MHLEN calculation do not take actual compiler- induced alignment and padding inside the complete struct mbuf into account. Accordingly appropriate attention is required when changing members of struct mbuf. Ideally one would calculate MLEN as (MSIZE - sizeof(((struct mbuf *)0)->m_hdr) but that doesn't work as the compiler refuses to operate on an as of yet incomplete structure. In particular ARM 32bit has more strict alignment requirements which caused 4 bytes of padding between m_hdr and pkthdr in struct mbuf because of the 64bit members in pkthdr. This wasn't picked up by MLEN and MHLEN causing an overflow of the mbuf provided data storage by overestimating its size. I386 didn't show this problem because it handles unaligned access just fine, albeit at a small performance penalty. On 64bit architectures the struct mbuf layout is 64bit aligned in all places. Reported by: Thomas Skibo <ThomasSkibo-at-sbcglobal-dot-net> Tested by: tuexen, ian, Thomas Skibo (extended patch) Sponsored by: The FreeBSD Foundation
2013-08-27 20:52:02 +00:00
/*
* Ensure the correct size of various mbuf parameters. It could be off due
* to compiler-induced padding and alignment artifacts.
*/
CTASSERT(MSIZE - offsetof(struct mbuf, m_dat) == MLEN);
CTASSERT(MSIZE - offsetof(struct mbuf, m_pktdat) == MHLEN);
In order to support ongoing work to implement variable-size mbufs, and more generally make it easier to extend 'struct mbuf in the future', make a number of changes to the data structure: - As we anticipate embedding mbufs headers within variable-size regions of memory in the future, change the definitions of byte arrays embedded in mbufs to be of size [0] rather than [MLEN] and [MHLEN]. In fact, the cxgbe driver already uses 'struct mbuf' on the front of other storage sizes, but we would like the global mbuf allocator do be able to do this as well. - Fold 'struct m_hdr' into 'struct mbuf' itself, eliminating a set of macros that aliased 'mh_foo' field names to 'm_foo' names such as 'm_next'. These present a particular problem as we would like to add new mbuf-header fields -- e.g., 'm_size' -- that, if similarly named via macros, would introduce collisions with many other variable names in the kernel. - Rename 'struct m_ext' to 'struct struct_m_ext' so that we can add compile-time assertions without bumping into the still-extant 'm_ext' macro. - Remove the MSIZE compile-time assertion for 'struct mbuf', but add new assertions for alignment of embedded data arrays (64-bit alignment even on 32-bit platforms), and for the sizes the mbuf header, packet header, and m_ext structure. - Document that these assertions exist in comments in mbuf.h. This change is not intended to cause (non-trivial) behavioural differences, but is a precursor to further mbuf-allocator work. Differential Revision: https://reviews.freebsd.org/D1483 Reviewed by: bz, gnn, np, glebius ("go ahead, I trust you") Sponsored by: EMC / Isilon Storage Division
2015-01-14 23:44:00 +00:00
/*
* mbuf data storage should be 64-bit aligned regardless of architectural
* pointer size; check this is the case with and without a packet header.
*/
CTASSERT(offsetof(struct mbuf, m_dat) % 8 == 0);
CTASSERT(offsetof(struct mbuf, m_pktdat) % 8 == 0);
/*
* While the specific values here don't matter too much (i.e., +/- a few
* words), we do want to ensure that changes to these values are carefully
* reasoned about and properly documented. This is especially the case as
* network-protocol and device-driver modules encode these layouts, and must
* be recompiled if the structures change. Check these values at compile time
* against the ones documented in comments in mbuf.h.
*
* NB: Possibly they should be documented there via #define's and not just
* comments.
*/
#if defined(__LP64__)
CTASSERT(offsetof(struct mbuf, m_dat) == 32);
CTASSERT(sizeof(struct pkthdr) == 56);
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
CTASSERT(sizeof(struct m_ext) == 160);
In order to support ongoing work to implement variable-size mbufs, and more generally make it easier to extend 'struct mbuf in the future', make a number of changes to the data structure: - As we anticipate embedding mbufs headers within variable-size regions of memory in the future, change the definitions of byte arrays embedded in mbufs to be of size [0] rather than [MLEN] and [MHLEN]. In fact, the cxgbe driver already uses 'struct mbuf' on the front of other storage sizes, but we would like the global mbuf allocator do be able to do this as well. - Fold 'struct m_hdr' into 'struct mbuf' itself, eliminating a set of macros that aliased 'mh_foo' field names to 'm_foo' names such as 'm_next'. These present a particular problem as we would like to add new mbuf-header fields -- e.g., 'm_size' -- that, if similarly named via macros, would introduce collisions with many other variable names in the kernel. - Rename 'struct m_ext' to 'struct struct_m_ext' so that we can add compile-time assertions without bumping into the still-extant 'm_ext' macro. - Remove the MSIZE compile-time assertion for 'struct mbuf', but add new assertions for alignment of embedded data arrays (64-bit alignment even on 32-bit platforms), and for the sizes the mbuf header, packet header, and m_ext structure. - Document that these assertions exist in comments in mbuf.h. This change is not intended to cause (non-trivial) behavioural differences, but is a precursor to further mbuf-allocator work. Differential Revision: https://reviews.freebsd.org/D1483 Reviewed by: bz, gnn, np, glebius ("go ahead, I trust you") Sponsored by: EMC / Isilon Storage Division
2015-01-14 23:44:00 +00:00
#else
CTASSERT(offsetof(struct mbuf, m_dat) == 24);
CTASSERT(sizeof(struct pkthdr) == 48);
[PowerPC] Fix powerpcspe build failure after r360569 On powerpcspe, vm_paddr_t is 64 bit despite it being a 32 bit platform. Adjust compile time assertion to compensate.
2020-05-07 17:58:07 +00:00
#if defined(__powerpc__) && defined(BOOKE)
/* PowerPC booke has 64-bit physical pointers. */
CTASSERT(sizeof(struct m_ext) == 184);
#else
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
CTASSERT(sizeof(struct m_ext) == 180);
In order to support ongoing work to implement variable-size mbufs, and more generally make it easier to extend 'struct mbuf in the future', make a number of changes to the data structure: - As we anticipate embedding mbufs headers within variable-size regions of memory in the future, change the definitions of byte arrays embedded in mbufs to be of size [0] rather than [MLEN] and [MHLEN]. In fact, the cxgbe driver already uses 'struct mbuf' on the front of other storage sizes, but we would like the global mbuf allocator do be able to do this as well. - Fold 'struct m_hdr' into 'struct mbuf' itself, eliminating a set of macros that aliased 'mh_foo' field names to 'm_foo' names such as 'm_next'. These present a particular problem as we would like to add new mbuf-header fields -- e.g., 'm_size' -- that, if similarly named via macros, would introduce collisions with many other variable names in the kernel. - Rename 'struct m_ext' to 'struct struct_m_ext' so that we can add compile-time assertions without bumping into the still-extant 'm_ext' macro. - Remove the MSIZE compile-time assertion for 'struct mbuf', but add new assertions for alignment of embedded data arrays (64-bit alignment even on 32-bit platforms), and for the sizes the mbuf header, packet header, and m_ext structure. - Document that these assertions exist in comments in mbuf.h. This change is not intended to cause (non-trivial) behavioural differences, but is a precursor to further mbuf-allocator work. Differential Revision: https://reviews.freebsd.org/D1483 Reviewed by: bz, gnn, np, glebius ("go ahead, I trust you") Sponsored by: EMC / Isilon Storage Division
2015-01-14 23:44:00 +00:00
#endif
[PowerPC] Fix powerpcspe build failure after r360569 On powerpcspe, vm_paddr_t is 64 bit despite it being a 32 bit platform. Adjust compile time assertion to compensate.
2020-05-07 17:58:07 +00:00
#endif
In order to support ongoing work to implement variable-size mbufs, and more generally make it easier to extend 'struct mbuf in the future', make a number of changes to the data structure: - As we anticipate embedding mbufs headers within variable-size regions of memory in the future, change the definitions of byte arrays embedded in mbufs to be of size [0] rather than [MLEN] and [MHLEN]. In fact, the cxgbe driver already uses 'struct mbuf' on the front of other storage sizes, but we would like the global mbuf allocator do be able to do this as well. - Fold 'struct m_hdr' into 'struct mbuf' itself, eliminating a set of macros that aliased 'mh_foo' field names to 'm_foo' names such as 'm_next'. These present a particular problem as we would like to add new mbuf-header fields -- e.g., 'm_size' -- that, if similarly named via macros, would introduce collisions with many other variable names in the kernel. - Rename 'struct m_ext' to 'struct struct_m_ext' so that we can add compile-time assertions without bumping into the still-extant 'm_ext' macro. - Remove the MSIZE compile-time assertion for 'struct mbuf', but add new assertions for alignment of embedded data arrays (64-bit alignment even on 32-bit platforms), and for the sizes the mbuf header, packet header, and m_ext structure. - Document that these assertions exist in comments in mbuf.h. This change is not intended to cause (non-trivial) behavioural differences, but is a precursor to further mbuf-allocator work. Differential Revision: https://reviews.freebsd.org/D1483 Reviewed by: bz, gnn, np, glebius ("go ahead, I trust you") Sponsored by: EMC / Isilon Storage Division
2015-01-14 23:44:00 +00:00
Use anonymous unions to add possibility to put mbufs into queue(3) STAILQs and SLISTs using the same structure field as good old m_next and m_nextpkt linkage occupy. New code is encouraged to use queue(3) macros, instead of implementing the wheel. However, better not to have a mixture of old style and queue(3) in one file or subsystem. Reviewed by: rwatson, rrs, rpaulo Differential Revision: D1499
2015-02-17 19:32:11 +00:00
/*
* Assert that the queue(3) macros produce code of the same size as an old
* plain pointer does.
*/
#ifdef INVARIANTS
Suppress a warning about m_assertbuf being unused. MFC after: 1 week
2017-01-15 03:53:20 +00:00
static struct mbuf __used m_assertbuf;
Use anonymous unions to add possibility to put mbufs into queue(3) STAILQs and SLISTs using the same structure field as good old m_next and m_nextpkt linkage occupy. New code is encouraged to use queue(3) macros, instead of implementing the wheel. However, better not to have a mixture of old style and queue(3) in one file or subsystem. Reviewed by: rwatson, rrs, rpaulo Differential Revision: D1499
2015-02-17 19:32:11 +00:00
CTASSERT(sizeof(m_assertbuf.m_slist) == sizeof(m_assertbuf.m_next));
CTASSERT(sizeof(m_assertbuf.m_stailq) == sizeof(m_assertbuf.m_next));
CTASSERT(sizeof(m_assertbuf.m_slistpkt) == sizeof(m_assertbuf.m_nextpkt));
CTASSERT(sizeof(m_assertbuf.m_stailqpkt) == sizeof(m_assertbuf.m_nextpkt));
#endif
Mandatory mbuf cluster reference counting and groundwork for UMA based jumbo 9k and jumbo 16k cluster support. All mbuf's with external storage attached are mandatory reference counted. For clusters and jumbo clusters UMA provides the refcnt storage directly. It does not have to be separatly allocated. Any other type of external storage gets its own refcnt allocated from an UMA mbuf refcnt zone instead of normal kernel malloc. The refcount API MEXT_ADD_REF() and MEXT_REM_REF() is no longer publically accessible. The proper m_* functions have to be used. mb_ctor_clust() and mb_dtor_clust() both handle normal 2K as well as 9k and 16k clusters. Clusters and jumbo clusters may be obtained without attaching it immideatly to an mbuf. This is for high performance cluster allocation in network drivers where mbufs are attached after the cluster has been filled. Tested by: rwatson Sponsored by: TCP/IP Optimizations Fundraise 2005
2005-11-02 16:20:36 +00:00
/*
Fix typos - remove duplicate "the". PR: bin/154928 Submitted by: Eitan Adler <lists at eitanadler.com> MFC after: 3 days
2011-02-21 09:01:34 +00:00
* Attach the cluster from *m to *n, set up m_ext in *n
Mandatory mbuf cluster reference counting and groundwork for UMA based jumbo 9k and jumbo 16k cluster support. All mbuf's with external storage attached are mandatory reference counted. For clusters and jumbo clusters UMA provides the refcnt storage directly. It does not have to be separatly allocated. Any other type of external storage gets its own refcnt allocated from an UMA mbuf refcnt zone instead of normal kernel malloc. The refcount API MEXT_ADD_REF() and MEXT_REM_REF() is no longer publically accessible. The proper m_* functions have to be used. mb_ctor_clust() and mb_dtor_clust() both handle normal 2K as well as 9k and 16k clusters. Clusters and jumbo clusters may be obtained without attaching it immideatly to an mbuf. This is for high performance cluster allocation in network drivers where mbufs are attached after the cluster has been filled. Tested by: rwatson Sponsored by: TCP/IP Optimizations Fundraise 2005
2005-11-02 16:20:36 +00:00
* and bump the refcount of the cluster.
*/
There are times when it would be really nice to have a record of the last few packets and/or state transitions from each TCP socket. That would help with narrowing down certain problems we see in the field that are hard to reproduce without understanding the history of how we got into a certain state. This change provides just that. It saves copies of the last N packets in a list in the tcpcb. When the tcpcb is destroyed, the list is freed. I thought this was likely to be more performance-friendly than saving copies of the tcpcb. Plus, with the packets, you should be able to reverse-engineer what happened to the tcpcb. To enable the feature, you will need to compile a kernel with the TCPPCAP option. Even then, the feature defaults to being deactivated. You can activate it by setting a positive value for the number of captured packets. You can do that on either a global basis or on a per-socket basis (via a setsockopt call). There is no way to get the packets out of the kernel other than using kmem or getting a coredump. I thought that would help some of the legal/privacy concerns regarding such a feature. However, it should be possible to add a future effort to export them in PCAP format. I tested this at low scale, and found that there were no mbuf leaks and the peak mbuf usage appeared to be unchanged with and without the feature. The main performance concern I can envision is the number of mbufs that would be used on systems with a large number of sockets. If you save five packets per direction per socket and have 3,000 sockets, that will consume at least 30,000 mbufs just to keep these packets. I tried to reduce the concerns associated with this by limiting the number of clusters (not mbufs) that could be used for this feature. Again, in my testing, that appears to work correctly. Differential Revision: D3100 Submitted by: Jonathan Looney <jlooney at juniper dot net> Reviewed by: gnn, hiren
2015-10-14 00:35:37 +00:00
void
New way to manage reference counting of mbuf external storage. The m_ext.ext_cnt pointer becomes a union. It can now hold the refcount value itself. To tell that m_ext.ext_flags flag EXT_FLAG_EMBREF is used. The first mbuf to attach a cluster stores the refcount. The further mbufs to reference the cluster point at refcount in the first mbuf. The first mbuf is freed only when the last reference is freed. The benefit over refcounts stored in separate slabs is that now refcounts of different, unrelated mbufs do not share a cache line. For EXT_EXTREF mbufs the zone_ext_refcnt is no longer needed, and m_extadd() becomes void, making widely used M_EXTADD macro safe. For EXT_SFBUF mbufs the sf_ext_ref() is removed, which was an optimization exactly against the cache aliasing problem with regular refcounting. Discussed with: rrs, rwatson, gnn, hiren, sbruno, np Reviewed by: rrs Differential Revision: https://reviews.freebsd.org/D5396 Sponsored by: Netflix
2016-03-01 00:17:14 +00:00
mb_dupcl(struct mbuf *n, struct mbuf *m)
Mandatory mbuf cluster reference counting and groundwork for UMA based jumbo 9k and jumbo 16k cluster support. All mbuf's with external storage attached are mandatory reference counted. For clusters and jumbo clusters UMA provides the refcnt storage directly. It does not have to be separatly allocated. Any other type of external storage gets its own refcnt allocated from an UMA mbuf refcnt zone instead of normal kernel malloc. The refcount API MEXT_ADD_REF() and MEXT_REM_REF() is no longer publically accessible. The proper m_* functions have to be used. mb_ctor_clust() and mb_dtor_clust() both handle normal 2K as well as 9k and 16k clusters. Clusters and jumbo clusters may be obtained without attaching it immideatly to an mbuf. This is for high performance cluster allocation in network drivers where mbufs are attached after the cluster has been filled. Tested by: rwatson Sponsored by: TCP/IP Optimizations Fundraise 2005
2005-11-02 16:20:36 +00:00
{
New way to manage reference counting of mbuf external storage. The m_ext.ext_cnt pointer becomes a union. It can now hold the refcount value itself. To tell that m_ext.ext_flags flag EXT_FLAG_EMBREF is used. The first mbuf to attach a cluster stores the refcount. The further mbufs to reference the cluster point at refcount in the first mbuf. The first mbuf is freed only when the last reference is freed. The benefit over refcounts stored in separate slabs is that now refcounts of different, unrelated mbufs do not share a cache line. For EXT_EXTREF mbufs the zone_ext_refcnt is no longer needed, and m_extadd() becomes void, making widely used M_EXTADD macro safe. For EXT_SFBUF mbufs the sf_ext_ref() is removed, which was an optimization exactly against the cache aliasing problem with regular refcounting. Discussed with: rrs, rwatson, gnn, hiren, sbruno, np Reviewed by: rrs Differential Revision: https://reviews.freebsd.org/D5396 Sponsored by: Netflix
2016-03-01 00:17:14 +00:00
volatile u_int *refcnt;
Improve reference counting of EXT_SFBUF pages attached to mbufs. o Do not use UMA refcount zone. The problem with this zone is that several refcounting words (16 on amd64) share the same cache line, and issueing atomic(9) updates on them creates cache line contention. Also, allocating and freeing them is extra CPU cycles. Instead, refcount the page directly via vm_page_wire() and the sfbuf via sf_buf_alloc(sf_buf_page(sf)) [1]. o Call refcounting/freeing function for EXT_SFBUF via direct function call, instead of function pointer. This removes barrier for CPU branch predictor. o Do not cleanup the mbuf to be freed in mb_free_ext(), merely to satisfy assertion in mb_dtor_mbuf(). Remove the assertion from mb_dtor_mbuf(). Use bcopy() instead of manual assignments to copy m_ext in mb_dupcl(). [1] This has some problems for now. Using sf_buf_alloc() merely to increase refcount is expensive, and is broken on sparc64. To be fixed. Sponsored by: Netflix Sponsored by: Nginx, Inc.
2014-07-11 19:40:50 +00:00
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
KASSERT(m->m_flags & (M_EXT|M_EXTPG),
("%s: M_EXT|M_EXTPG not set on %p", __func__, m));
KASSERT(!(n->m_flags & (M_EXT|M_EXTPG)),
("%s: M_EXT|M_EXTPG set on %p", __func__, n));
Improve reference counting of EXT_SFBUF pages attached to mbufs. o Do not use UMA refcount zone. The problem with this zone is that several refcounting words (16 on amd64) share the same cache line, and issueing atomic(9) updates on them creates cache line contention. Also, allocating and freeing them is extra CPU cycles. Instead, refcount the page directly via vm_page_wire() and the sfbuf via sf_buf_alloc(sf_buf_page(sf)) [1]. o Call refcounting/freeing function for EXT_SFBUF via direct function call, instead of function pointer. This removes barrier for CPU branch predictor. o Do not cleanup the mbuf to be freed in mb_free_ext(), merely to satisfy assertion in mb_dtor_mbuf(). Remove the assertion from mb_dtor_mbuf(). Use bcopy() instead of manual assignments to copy m_ext in mb_dupcl(). [1] This has some problems for now. Using sf_buf_alloc() merely to increase refcount is expensive, and is broken on sparc64. To be fixed. Sponsored by: Netflix Sponsored by: Nginx, Inc.
2014-07-11 19:40:50 +00:00
In mb_dupcl() don't copy full m_ext, to avoid cache miss. Respectively, in mb_free_ext() always use fields from the original refcount holding mbuf (see. r296242) mbuf. Cuts another cache miss from mb_free_ext(). However, treat EXT_EXTREF mbufs differently, since they are different - they don't have a refcount holding mbuf. Provide longer comments in m_ext declaration to explain this change and change from r296242. In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 20:51:58 +00:00
/*
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
* Cache access optimization.
*
* o Regular M_EXT storage doesn't need full copy of m_ext, since
* the holder of the 'ext_count' is responsible to carry the free
* routine and its arguments.
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
* o M_EXTPG data is split between main part of mbuf and m_ext, the
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
* main part is copied in full, the m_ext part is similar to M_EXT.
* o EXT_EXTREF, where 'ext_cnt' doesn't point into mbuf at all, is
* special - it needs full copy of m_ext into each mbuf, since any
* copy could end up as the last to free.
In mb_dupcl() don't copy full m_ext, to avoid cache miss. Respectively, in mb_free_ext() always use fields from the original refcount holding mbuf (see. r296242) mbuf. Cuts another cache miss from mb_free_ext(). However, treat EXT_EXTREF mbufs differently, since they are different - they don't have a refcount holding mbuf. Provide longer comments in m_ext declaration to explain this change and change from r296242. In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 20:51:58 +00:00
*/
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
if (m->m_flags & M_EXTPG) {
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
bcopy(&m->m_epg_startcopy, &n->m_epg_startcopy,
__rangeof(struct mbuf, m_epg_startcopy, m_epg_endcopy));
bcopy(&m->m_ext, &n->m_ext, m_epg_ext_copylen);
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
} else if (m->m_ext.ext_type == EXT_EXTREF)
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
bcopy(&m->m_ext, &n->m_ext, sizeof(struct m_ext));
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
else
In mb_dupcl() don't copy full m_ext, to avoid cache miss. Respectively, in mb_free_ext() always use fields from the original refcount holding mbuf (see. r296242) mbuf. Cuts another cache miss from mb_free_ext(). However, treat EXT_EXTREF mbufs differently, since they are different - they don't have a refcount holding mbuf. Provide longer comments in m_ext declaration to explain this change and change from r296242. In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 20:51:58 +00:00
bcopy(&m->m_ext, &n->m_ext, m_ext_copylen);
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
n->m_flags |= m->m_flags & (M_RDONLY | M_EXT | M_EXTPG);
New way to manage reference counting of mbuf external storage. The m_ext.ext_cnt pointer becomes a union. It can now hold the refcount value itself. To tell that m_ext.ext_flags flag EXT_FLAG_EMBREF is used. The first mbuf to attach a cluster stores the refcount. The further mbufs to reference the cluster point at refcount in the first mbuf. The first mbuf is freed only when the last reference is freed. The benefit over refcounts stored in separate slabs is that now refcounts of different, unrelated mbufs do not share a cache line. For EXT_EXTREF mbufs the zone_ext_refcnt is no longer needed, and m_extadd() becomes void, making widely used M_EXTADD macro safe. For EXT_SFBUF mbufs the sf_ext_ref() is removed, which was an optimization exactly against the cache aliasing problem with regular refcounting. Discussed with: rrs, rwatson, gnn, hiren, sbruno, np Reviewed by: rrs Differential Revision: https://reviews.freebsd.org/D5396 Sponsored by: Netflix
2016-03-01 00:17:14 +00:00
/* See if this is the mbuf that holds the embedded refcount. */
if (m->m_ext.ext_flags & EXT_FLAG_EMBREF) {
refcnt = n->m_ext.ext_cnt = &m->m_ext.ext_count;
n->m_ext.ext_flags &= ~EXT_FLAG_EMBREF;
} else {
Improve reference counting of EXT_SFBUF pages attached to mbufs. o Do not use UMA refcount zone. The problem with this zone is that several refcounting words (16 on amd64) share the same cache line, and issueing atomic(9) updates on them creates cache line contention. Also, allocating and freeing them is extra CPU cycles. Instead, refcount the page directly via vm_page_wire() and the sfbuf via sf_buf_alloc(sf_buf_page(sf)) [1]. o Call refcounting/freeing function for EXT_SFBUF via direct function call, instead of function pointer. This removes barrier for CPU branch predictor. o Do not cleanup the mbuf to be freed in mb_free_ext(), merely to satisfy assertion in mb_dtor_mbuf(). Remove the assertion from mb_dtor_mbuf(). Use bcopy() instead of manual assignments to copy m_ext in mb_dupcl(). [1] This has some problems for now. Using sf_buf_alloc() merely to increase refcount is expensive, and is broken on sparc64. To be fixed. Sponsored by: Netflix Sponsored by: Nginx, Inc.
2014-07-11 19:40:50 +00:00
KASSERT(m->m_ext.ext_cnt != NULL,
("%s: no refcounting pointer on %p", __func__, m));
New way to manage reference counting of mbuf external storage. The m_ext.ext_cnt pointer becomes a union. It can now hold the refcount value itself. To tell that m_ext.ext_flags flag EXT_FLAG_EMBREF is used. The first mbuf to attach a cluster stores the refcount. The further mbufs to reference the cluster point at refcount in the first mbuf. The first mbuf is freed only when the last reference is freed. The benefit over refcounts stored in separate slabs is that now refcounts of different, unrelated mbufs do not share a cache line. For EXT_EXTREF mbufs the zone_ext_refcnt is no longer needed, and m_extadd() becomes void, making widely used M_EXTADD macro safe. For EXT_SFBUF mbufs the sf_ext_ref() is removed, which was an optimization exactly against the cache aliasing problem with regular refcounting. Discussed with: rrs, rwatson, gnn, hiren, sbruno, np Reviewed by: rrs Differential Revision: https://reviews.freebsd.org/D5396 Sponsored by: Netflix
2016-03-01 00:17:14 +00:00
refcnt = m->m_ext.ext_cnt;
Improve reference counting of EXT_SFBUF pages attached to mbufs. o Do not use UMA refcount zone. The problem with this zone is that several refcounting words (16 on amd64) share the same cache line, and issueing atomic(9) updates on them creates cache line contention. Also, allocating and freeing them is extra CPU cycles. Instead, refcount the page directly via vm_page_wire() and the sfbuf via sf_buf_alloc(sf_buf_page(sf)) [1]. o Call refcounting/freeing function for EXT_SFBUF via direct function call, instead of function pointer. This removes barrier for CPU branch predictor. o Do not cleanup the mbuf to be freed in mb_free_ext(), merely to satisfy assertion in mb_dtor_mbuf(). Remove the assertion from mb_dtor_mbuf(). Use bcopy() instead of manual assignments to copy m_ext in mb_dupcl(). [1] This has some problems for now. Using sf_buf_alloc() merely to increase refcount is expensive, and is broken on sparc64. To be fixed. Sponsored by: Netflix Sponsored by: Nginx, Inc.
2014-07-11 19:40:50 +00:00
}
New way to manage reference counting of mbuf external storage. The m_ext.ext_cnt pointer becomes a union. It can now hold the refcount value itself. To tell that m_ext.ext_flags flag EXT_FLAG_EMBREF is used. The first mbuf to attach a cluster stores the refcount. The further mbufs to reference the cluster point at refcount in the first mbuf. The first mbuf is freed only when the last reference is freed. The benefit over refcounts stored in separate slabs is that now refcounts of different, unrelated mbufs do not share a cache line. For EXT_EXTREF mbufs the zone_ext_refcnt is no longer needed, and m_extadd() becomes void, making widely used M_EXTADD macro safe. For EXT_SFBUF mbufs the sf_ext_ref() is removed, which was an optimization exactly against the cache aliasing problem with regular refcounting. Discussed with: rrs, rwatson, gnn, hiren, sbruno, np Reviewed by: rrs Differential Revision: https://reviews.freebsd.org/D5396 Sponsored by: Netflix
2016-03-01 00:17:14 +00:00
if (*refcnt == 1)
*refcnt += 1;
else
atomic_add_int(refcnt, 1);
Mandatory mbuf cluster reference counting and groundwork for UMA based jumbo 9k and jumbo 16k cluster support. All mbuf's with external storage attached are mandatory reference counted. For clusters and jumbo clusters UMA provides the refcnt storage directly. It does not have to be separatly allocated. Any other type of external storage gets its own refcnt allocated from an UMA mbuf refcnt zone instead of normal kernel malloc. The refcount API MEXT_ADD_REF() and MEXT_REM_REF() is no longer publically accessible. The proper m_* functions have to be used. mb_ctor_clust() and mb_dtor_clust() both handle normal 2K as well as 9k and 16k clusters. Clusters and jumbo clusters may be obtained without attaching it immideatly to an mbuf. This is for high performance cluster allocation in network drivers where mbufs are attached after the cluster has been filled. Tested by: rwatson Sponsored by: TCP/IP Optimizations Fundraise 2005
2005-11-02 16:20:36 +00:00
}
Fix leak in tcp_lro_rx. Simply clearing M_PKTHDR isn't enough, any tags hanging off the header need to be freed too. Differential Revision: https://reviews.freebsd.org/D2708 Reviewed by: ae@, hiren@
2015-06-30 17:19:58 +00:00
void
m_demote_pkthdr(struct mbuf *m)
{
M_ASSERTPKTHDR(m);
m_tag_delete_chain(m, NULL);
m->m_flags &= ~M_PKTHDR;
bzero(&m->m_pkthdr, sizeof(struct pkthdr));
}
Add m_demote(struct mbuf *m, int all) to clean up mbuf (chain) from any tags and packet headers. If "all" is set then the first mbuf in the chain will be cleaned too. This function is used before an mbuf, that arrived as packet with m->flags & M_PKTHDR, is appended to an mbuf chain using m->m_next (not m->m_nextpkt). Reviewed by: glebius
2005-08-29 19:45:39 +00:00
/*
* Clean up mbuf (chain) from any tags and packet headers.
Unbreak m_demote() and put back the 'all' flag. Without it we cannot correctly test for m_nextpkt in an mbuf chain.
2005-08-30 21:14:30 +00:00
* If "all" is set then the first mbuf in the chain will be
* cleaned too.
Add m_demote(struct mbuf *m, int all) to clean up mbuf (chain) from any tags and packet headers. If "all" is set then the first mbuf in the chain will be cleaned too. This function is used before an mbuf, that arrived as packet with m->flags & M_PKTHDR, is appended to an mbuf chain using m->m_next (not m->m_nextpkt). Reviewed by: glebius
2005-08-29 19:45:39 +00:00
*/
void
Merge from projects/sendfile: extend protocols API to support sending not ready data: o Add new flag to pru_send() flags - PRUS_NOTREADY. o Add new protocol method pru_ready(). Sponsored by: Nginx, Inc. Sponsored by: Netflix
2014-11-30 13:24:21 +00:00
m_demote(struct mbuf *m0, int all, int flags)
Add m_demote(struct mbuf *m, int all) to clean up mbuf (chain) from any tags and packet headers. If "all" is set then the first mbuf in the chain will be cleaned too. This function is used before an mbuf, that arrived as packet with m->flags & M_PKTHDR, is appended to an mbuf chain using m->m_next (not m->m_nextpkt). Reviewed by: glebius
2005-08-29 19:45:39 +00:00
{
struct mbuf *m;
for (m = all ? m0 : m0->m_next; m != NULL; m = m->m_next) {
Change a very strange code in m_demote() to simple assertion. Sponsored by: Nginx, Inc.
2014-09-04 19:27:30 +00:00
KASSERT(m->m_nextpkt == NULL, ("%s: m_nextpkt in m %p, m0 %p",
__func__, m, m0));
Fix leak in tcp_lro_rx. Simply clearing M_PKTHDR isn't enough, any tags hanging off the header need to be freed too. Differential Revision: https://reviews.freebsd.org/D2708 Reviewed by: ae@, hiren@
2015-06-30 17:19:58 +00:00
if (m->m_flags & M_PKTHDR)
m_demote_pkthdr(m);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
m->m_flags = m->m_flags & (M_EXT | M_RDONLY | M_NOFREE |
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
M_EXTPG | flags);
Add m_demote(struct mbuf *m, int all) to clean up mbuf (chain) from any tags and packet headers. If "all" is set then the first mbuf in the chain will be cleaned too. This function is used before an mbuf, that arrived as packet with m->flags & M_PKTHDR, is appended to an mbuf chain using m->m_next (not m->m_nextpkt). Reviewed by: glebius
2005-08-29 19:45:39 +00:00
}
}
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
/*
Changes and cleanups to m_sanity(): o for() instead of while() looping over mbuf chain o paren's around all flag checks o more verbose function and purpose description o some more style changes Based on feedback from: sam
2005-08-30 21:31:42 +00:00
* Sanity checks on mbuf (chain) for use in KASSERT() and general
* debugging.
* Returns 0 or panics when bad and 1 on all tests passed.
* Sanitize, 0 to run M_SANITY_ACTION, 1 to garble things so they
* blow up later.
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
*/
int
m_sanity(struct mbuf *m0, int sanitize)
{
struct mbuf *m;
caddr_t a, b;
int pktlen = 0;
remove now invalid check from m_sanity panic on m_sanity check failure with INVARIANTS
2007-04-14 20:19:16 +00:00
#ifdef INVARIANTS
#define M_SANITY_ACTION(s) panic("mbuf %p: " s, m)
Whitespace only.
2014-05-30 08:22:58 +00:00
#else
remove now invalid check from m_sanity panic on m_sanity check failure with INVARIANTS
2007-04-14 20:19:16 +00:00
#define M_SANITY_ACTION(s) printf("mbuf %p: " s, m)
#endif
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
Changes and cleanups to m_sanity(): o for() instead of while() looping over mbuf chain o paren's around all flag checks o more verbose function and purpose description o some more style changes Based on feedback from: sam
2005-08-30 21:31:42 +00:00
for (m = m0; m != NULL; m = m->m_next) {
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
/*
* Basic pointer checks. If any of these fails then some
* unrelated kernel memory before or after us is trashed.
* No way to recover from that.
*/
Replace hand-crafted versions of M_SIZE() and M_START() in uipc_mbuf.c with calls to the centralised macros, reducing direct use of MLEN and MHLEN. Differential Revision: https://reviews.freebsd.org/D1444 Reviewed by: bz Sponsored by: EMC / Isilon Storage Division
2015-01-08 11:16:21 +00:00
a = M_START(m);
b = a + M_SIZE(m);
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
if ((caddr_t)m->m_data < a)
M_SANITY_ACTION("m_data outside mbuf data range left");
if ((caddr_t)m->m_data > b)
M_SANITY_ACTION("m_data outside mbuf data range right");
if ((caddr_t)m->m_data + m->m_len > b)
M_SANITY_ACTION("m_data + m_len exeeds mbuf space");
/* m->m_nextpkt may only be set on first mbuf in chain. */
Changes and cleanups to m_sanity(): o for() instead of while() looping over mbuf chain o paren's around all flag checks o more verbose function and purpose description o some more style changes Based on feedback from: sam
2005-08-30 21:31:42 +00:00
if (m != m0 && m->m_nextpkt != NULL) {
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
if (sanitize) {
m_freem(m->m_nextpkt);
m->m_nextpkt = (struct mbuf *)0xDEADC0DE;
} else
M_SANITY_ACTION("m->m_nextpkt on in-chain mbuf");
}
/* packet length (not mbuf length!) calculation */
if (m0->m_flags & M_PKTHDR)
pktlen += m->m_len;
/* m_tags may only be attached to first mbuf in chain. */
if (m != m0 && m->m_flags & M_PKTHDR &&
!SLIST_EMPTY(&m->m_pkthdr.tags)) {
if (sanitize) {
m_tag_delete_chain(m, NULL);
/* put in 0xDEADC0DE perhaps? */
Changes and cleanups to m_sanity(): o for() instead of while() looping over mbuf chain o paren's around all flag checks o more verbose function and purpose description o some more style changes Based on feedback from: sam
2005-08-30 21:31:42 +00:00
} else
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
M_SANITY_ACTION("m_tags on in-chain mbuf");
}
/* M_PKTHDR may only be set on first mbuf in chain */
if (m != m0 && m->m_flags & M_PKTHDR) {
if (sanitize) {
bzero(&m->m_pkthdr, sizeof(m->m_pkthdr));
m->m_flags &= ~M_PKTHDR;
/* put in 0xDEADCODE and leave hdr flag in */
} else
M_SANITY_ACTION("M_PKTHDR on in-chain mbuf");
}
}
Changes and cleanups to m_sanity(): o for() instead of while() looping over mbuf chain o paren's around all flag checks o more verbose function and purpose description o some more style changes Based on feedback from: sam
2005-08-30 21:31:42 +00:00
m = m0;
if (pktlen && pktlen != m->m_pkthdr.len) {
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
if (sanitize)
Changes and cleanups to m_sanity(): o for() instead of while() looping over mbuf chain o paren's around all flag checks o more verbose function and purpose description o some more style changes Based on feedback from: sam
2005-08-30 21:31:42 +00:00
m->m_pkthdr.len = 0;
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
else
M_SANITY_ACTION("m_pkthdr.len != mbuf chain length");
}
return 1;
Changes and cleanups to m_sanity(): o for() instead of while() looping over mbuf chain o paren's around all flag checks o more verbose function and purpose description o some more style changes Based on feedback from: sam
2005-08-30 21:31:42 +00:00
#undef M_SANITY_ACTION
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
}
o Gather all mbuf(9) allocation functions into kern_mbuf.c, and all mbuf(9) manipulation functions into uipc_mbuf.c. This looks like the initial intent, but had diffused in the last decade. o Gather all declarations in mbuf.h in one place and sort them. o Uninline m_clget() and m_cljget(). There are no functional changes in this patch. The patch comes from a larger version, where all mbuf(9) allocation was uninlined, which allowed to make mbuf(9) UMA zones private to kern_mbuf.c. The performance impact of the total uninlining is still unclear, so we are holding on now with larger version. Together with: melifaro, olivier
2016-02-11 21:32:23 +00:00
/*
* Non-inlined part of m_init().
*/
int
m_pkthdr_init(struct mbuf *m, int how)
{
#ifdef MAC
int error;
#endif
m->m_data = m->m_pktdat;
bzero(&m->m_pkthdr, sizeof(m->m_pkthdr));
Track TCP connection's NUMA domain in the inpcb Drivers can now pass up numa domain information via the mbuf numa domain field. This information is then used by TCP syncache_socket() to associate that information with the inpcb. The domain information is then fed back into transmitted mbufs in ip{6}_output(). This mechanism is nearly identical to what is done to track RSS hash values in the inp_flowid. Follow on changes will use this information for lacp egress port selection, binding TCP pacers to the appropriate NUMA domain, etc. Reviewed by: markj, kib, slavash, bz, scottl, jtl, tuexen Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20028
2019-04-25 15:37:28 +00:00
#ifdef NUMA
m->m_pkthdr.numa_domain = M_NODOM;
#endif
o Gather all mbuf(9) allocation functions into kern_mbuf.c, and all mbuf(9) manipulation functions into uipc_mbuf.c. This looks like the initial intent, but had diffused in the last decade. o Gather all declarations in mbuf.h in one place and sort them. o Uninline m_clget() and m_cljget(). There are no functional changes in this patch. The patch comes from a larger version, where all mbuf(9) allocation was uninlined, which allowed to make mbuf(9) UMA zones private to kern_mbuf.c. The performance impact of the total uninlining is still unclear, so we are holding on now with larger version. Together with: melifaro, olivier
2016-02-11 21:32:23 +00:00
#ifdef MAC
/* If the label init fails, fail the alloc */
error = mac_mbuf_init(m, how);
if (error)
return (error);
#endif
return (0);
}
Add m_sanity(struct mbuf *m, int sanitize) to do some heavy sanity checking on mbuf's and mbuf chains. Set sanitize to 1 to garble illegal things and have them blow up later when used/accessed. m_sanity()'s main purpose is for KASSERT()'s and debugging of non- kosher mbuf manipulation (of which we have a number of). Reviewed by: glebius
2005-08-29 19:58:56 +00:00
Make M_COPY_PKTHDR() macro into a wrapper for a m_copy_pkthdr() function. This permits conditionally compiled extensions to the packet header copying semantic, such as extensions to copy MAC labels. Reviewed by: bmilekic Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
2002-07-30 18:28:58 +00:00
/*
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
* "Move" mbuf pkthdr from "from" to "to".
Make M_COPY_PKTHDR() macro into a wrapper for a m_copy_pkthdr() function. This permits conditionally compiled extensions to the packet header copying semantic, such as extensions to copy MAC labels. Reviewed by: bmilekic Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
2002-07-30 18:28:58 +00:00
* "from" must have M_PKTHDR set, and "to" must be empty.
*/
void
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
m_move_pkthdr(struct mbuf *to, struct mbuf *from)
Make M_COPY_PKTHDR() macro into a wrapper for a m_copy_pkthdr() function. This permits conditionally compiled extensions to the packet header copying semantic, such as extensions to copy MAC labels. Reviewed by: bmilekic Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
2002-07-30 18:28:58 +00:00
{
#if 0
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
/* see below for why these are not enabled */
Introduce an M_ASSERTPKTHDR() macro which performs the very common task of asserting that an mbuf has a packet header. Use it instead of hand- rolled versions wherever applicable. Submitted by: Hiten Pandya <hiten@unixdaemons.com>
2003-04-08 14:25:47 +00:00
M_ASSERTPKTHDR(to);
Move MAC label storage for mbufs into m_tags from the m_pkthdr structure, returning some additional room in the first mbuf in a chain, and avoiding feature-specific contents in the mbuf header. To do this: - Modify mbuf_to_label() to extract the tag, returning NULL if not found. - Introduce mac_init_mbuf_tag() which does most of the work mac_init_mbuf() used to do, except on an m_tag rather than an mbuf. - Scale back mac_init_mbuf() to perform m_tag allocation and invoke mac_init_mbuf_tag(). - Replace mac_destroy_mbuf() with mac_destroy_mbuf_tag(), since m_tag's are now GC'd deep in the m_tag/mbuf code rather than at a higher level when mbufs are directly free()'d. - Add mac_copy_mbuf_tag() to support m_copy_pkthdr() and related notions. - Generally change all references to mbuf labels so that they use mbuf_to_label() rather than &mbuf->m_pkthdr.label. This required no changes in the MAC policies (yay!). - Tweak mbuf release routines to not call mac_destroy_mbuf(), tag destruction takes care of it for us now. - Remove MAC magic from m_copy_pkthdr() and m_move_pkthdr() -- the existing m_tag support does all this for us. Note that we can no longer just zero the m_tag list on the target mbuf, rather, we have to delete the chain because m_tag's will already be hung off freshly allocated mbuf's. - Tweak m_tag copying routines so that if we're copying a MAC m_tag, we don't do a binary copy, rather, we initialize the new storage and do a deep copy of the label. - Remove use of MAC_FLAG_INITIALIZED in a few bizarre places having to do with mbuf header copies previously. - When an mbuf is copied in ip_input(), we no longer need to explicitly copy the label because it will get handled by the m_tag code now. - No longer any weird handling of MAC labels in if_loop.c during header copies. - Add MPC_LOADTIME_FLAG_LABELMBUFS flag to Biba, MLS, mac_test. In mac_test, handle the label==NULL case, since it can be dynamically loaded. In order to improve performance with this change, introduce the notion of "lazy MAC label allocation" -- only allocate m_tag storage for MAC labels if we're running with a policy that uses MAC labels on mbufs. Policies declare this intent by setting the MPC_LOADTIME_FLAG_LABELMBUFS flag in their load-time flags field during declaration. Note: this opens up the possibility of post-boot policy modules getting back NULL slot entries even though they have policy invariants of non-NULL slot entries, as the policy might have been loaded after the mbuf was allocated, leaving the mbuf without label storage. Policies that cannot handle this case must be declared as NOTLATE, or must be modified. - mac_labelmbufs holds the current cumulative status as to whether any policies require mbuf labeling or not. This is updated whenever the active policy set changes by the function mac_policy_updateflags(). The function iterates the list and checks whether any have the flag set. Write access to this variable is protected by the policy list; read access is currently not protected for performance reasons. This might change if it causes problems. - Add MAC_POLICY_LIST_ASSERT_EXCLUSIVE() to permit the flags update function to assert appropriate locks. - This makes allocation in mac_init_mbuf() conditional on the flag. Reviewed by: sam Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2003-04-14 20:39:06 +00:00
/* Note: with MAC, this may not be a good assertion. */
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
KASSERT(SLIST_EMPTY(&to->m_pkthdr.tags),
("m_move_pkthdr: to has tags"));
Introduce support for Mandatory Access Control and extensible kernel access control. Invoke additional MAC entry points when an mbuf packet header is copied to another mbuf: release the old label if any, reinitialize the new header, and ask the MAC framework to copy the header label data. Note that this requires a potential allocation operation, but m_copy_pkthdr() is not permitted to fail, so we must block. Since we now use interrupt threads, this is possible, but not desirable. Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
2002-07-31 01:51:34 +00:00
#endif
#ifdef MAC
Move MAC label storage for mbufs into m_tags from the m_pkthdr structure, returning some additional room in the first mbuf in a chain, and avoiding feature-specific contents in the mbuf header. To do this: - Modify mbuf_to_label() to extract the tag, returning NULL if not found. - Introduce mac_init_mbuf_tag() which does most of the work mac_init_mbuf() used to do, except on an m_tag rather than an mbuf. - Scale back mac_init_mbuf() to perform m_tag allocation and invoke mac_init_mbuf_tag(). - Replace mac_destroy_mbuf() with mac_destroy_mbuf_tag(), since m_tag's are now GC'd deep in the m_tag/mbuf code rather than at a higher level when mbufs are directly free()'d. - Add mac_copy_mbuf_tag() to support m_copy_pkthdr() and related notions. - Generally change all references to mbuf labels so that they use mbuf_to_label() rather than &mbuf->m_pkthdr.label. This required no changes in the MAC policies (yay!). - Tweak mbuf release routines to not call mac_destroy_mbuf(), tag destruction takes care of it for us now. - Remove MAC magic from m_copy_pkthdr() and m_move_pkthdr() -- the existing m_tag support does all this for us. Note that we can no longer just zero the m_tag list on the target mbuf, rather, we have to delete the chain because m_tag's will already be hung off freshly allocated mbuf's. - Tweak m_tag copying routines so that if we're copying a MAC m_tag, we don't do a binary copy, rather, we initialize the new storage and do a deep copy of the label. - Remove use of MAC_FLAG_INITIALIZED in a few bizarre places having to do with mbuf header copies previously. - When an mbuf is copied in ip_input(), we no longer need to explicitly copy the label because it will get handled by the m_tag code now. - No longer any weird handling of MAC labels in if_loop.c during header copies. - Add MPC_LOADTIME_FLAG_LABELMBUFS flag to Biba, MLS, mac_test. In mac_test, handle the label==NULL case, since it can be dynamically loaded. In order to improve performance with this change, introduce the notion of "lazy MAC label allocation" -- only allocate m_tag storage for MAC labels if we're running with a policy that uses MAC labels on mbufs. Policies declare this intent by setting the MPC_LOADTIME_FLAG_LABELMBUFS flag in their load-time flags field during declaration. Note: this opens up the possibility of post-boot policy modules getting back NULL slot entries even though they have policy invariants of non-NULL slot entries, as the policy might have been loaded after the mbuf was allocated, leaving the mbuf without label storage. Policies that cannot handle this case must be declared as NOTLATE, or must be modified. - mac_labelmbufs holds the current cumulative status as to whether any policies require mbuf labeling or not. This is updated whenever the active policy set changes by the function mac_policy_updateflags(). The function iterates the list and checks whether any have the flag set. Write access to this variable is protected by the policy list; read access is currently not protected for performance reasons. This might change if it causes problems. - Add MAC_POLICY_LIST_ASSERT_EXCLUSIVE() to permit the flags update function to assert appropriate locks. - This makes allocation in mac_init_mbuf() conditional on the flag. Reviewed by: sam Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2003-04-14 20:39:06 +00:00
/*
* XXXMAC: It could be this should also occur for non-MAC?
*/
Introduce support for Mandatory Access Control and extensible kernel access control. Invoke additional MAC entry points when an mbuf packet header is copied to another mbuf: release the old label if any, reinitialize the new header, and ask the MAC framework to copy the header label data. Note that this requires a potential allocation operation, but m_copy_pkthdr() is not permitted to fail, so we must block. Since we now use interrupt threads, this is possible, but not desirable. Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
2002-07-31 01:51:34 +00:00
if (to->m_flags & M_PKTHDR)
Move MAC label storage for mbufs into m_tags from the m_pkthdr structure, returning some additional room in the first mbuf in a chain, and avoiding feature-specific contents in the mbuf header. To do this: - Modify mbuf_to_label() to extract the tag, returning NULL if not found. - Introduce mac_init_mbuf_tag() which does most of the work mac_init_mbuf() used to do, except on an m_tag rather than an mbuf. - Scale back mac_init_mbuf() to perform m_tag allocation and invoke mac_init_mbuf_tag(). - Replace mac_destroy_mbuf() with mac_destroy_mbuf_tag(), since m_tag's are now GC'd deep in the m_tag/mbuf code rather than at a higher level when mbufs are directly free()'d. - Add mac_copy_mbuf_tag() to support m_copy_pkthdr() and related notions. - Generally change all references to mbuf labels so that they use mbuf_to_label() rather than &mbuf->m_pkthdr.label. This required no changes in the MAC policies (yay!). - Tweak mbuf release routines to not call mac_destroy_mbuf(), tag destruction takes care of it for us now. - Remove MAC magic from m_copy_pkthdr() and m_move_pkthdr() -- the existing m_tag support does all this for us. Note that we can no longer just zero the m_tag list on the target mbuf, rather, we have to delete the chain because m_tag's will already be hung off freshly allocated mbuf's. - Tweak m_tag copying routines so that if we're copying a MAC m_tag, we don't do a binary copy, rather, we initialize the new storage and do a deep copy of the label. - Remove use of MAC_FLAG_INITIALIZED in a few bizarre places having to do with mbuf header copies previously. - When an mbuf is copied in ip_input(), we no longer need to explicitly copy the label because it will get handled by the m_tag code now. - No longer any weird handling of MAC labels in if_loop.c during header copies. - Add MPC_LOADTIME_FLAG_LABELMBUFS flag to Biba, MLS, mac_test. In mac_test, handle the label==NULL case, since it can be dynamically loaded. In order to improve performance with this change, introduce the notion of "lazy MAC label allocation" -- only allocate m_tag storage for MAC labels if we're running with a policy that uses MAC labels on mbufs. Policies declare this intent by setting the MPC_LOADTIME_FLAG_LABELMBUFS flag in their load-time flags field during declaration. Note: this opens up the possibility of post-boot policy modules getting back NULL slot entries even though they have policy invariants of non-NULL slot entries, as the policy might have been loaded after the mbuf was allocated, leaving the mbuf without label storage. Policies that cannot handle this case must be declared as NOTLATE, or must be modified. - mac_labelmbufs holds the current cumulative status as to whether any policies require mbuf labeling or not. This is updated whenever the active policy set changes by the function mac_policy_updateflags(). The function iterates the list and checks whether any have the flag set. Write access to this variable is protected by the policy list; read access is currently not protected for performance reasons. This might change if it causes problems. - Add MAC_POLICY_LIST_ASSERT_EXCLUSIVE() to permit the flags update function to assert appropriate locks. - This makes allocation in mac_init_mbuf() conditional on the flag. Reviewed by: sam Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2003-04-14 20:39:06 +00:00
m_tag_delete_chain(to, NULL);
Make M_COPY_PKTHDR() macro into a wrapper for a m_copy_pkthdr() function. This permits conditionally compiled extensions to the packet header copying semantic, such as extensions to copy MAC labels. Reviewed by: bmilekic Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
2002-07-30 18:28:58 +00:00
#endif
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
to->m_flags = (from->m_flags & M_COPYFLAGS) |
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
(to->m_flags & (M_EXT | M_EXTPG));
allow the destination of m_move_pkthdr to have external storage (e.g. a cluster) Glanced at by: rwatson, silby
2005-03-08 17:52:01 +00:00
if ((to->m_flags & M_EXT) == 0)
to->m_data = to->m_pktdat;
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
to->m_pkthdr = from->m_pkthdr; /* especially tags */
SLIST_INIT(&from->m_pkthdr.tags); /* purge tags from src */
from->m_flags &= ~M_PKTHDR;
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
if (from->m_pkthdr.csum_flags & CSUM_SND_TAG) {
from->m_pkthdr.csum_flags &= ~CSUM_SND_TAG;
from->m_pkthdr.snd_tag = NULL;
}
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
}
/*
* Duplicate "from"'s mbuf pkthdr in "to".
* "from" must have M_PKTHDR set, and "to" must be empty.
* In particular, this does a deep copy of the packet tags.
*/
int
Add const-qualifiers for source mbuf argument in m_dup(), m_copym(), m_dup_pkthdr() and m_tag_copy_chain().
2015-08-08 15:50:46 +00:00
m_dup_pkthdr(struct mbuf *to, const struct mbuf *from, int how)
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
{
#if 0
/*
* The mbuf allocator only initializes the pkthdr
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
* when the mbuf is allocated with m_gethdr(). Many users
* (e.g. m_copy*, m_prepend) use m_get() and then
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
* smash the pkthdr as needed causing these
* assertions to trip. For now just disable them.
*/
Introduce an M_ASSERTPKTHDR() macro which performs the very common task of asserting that an mbuf has a packet header. Use it instead of hand- rolled versions wherever applicable. Submitted by: Hiten Pandya <hiten@unixdaemons.com>
2003-04-08 14:25:47 +00:00
M_ASSERTPKTHDR(to);
Move MAC label storage for mbufs into m_tags from the m_pkthdr structure, returning some additional room in the first mbuf in a chain, and avoiding feature-specific contents in the mbuf header. To do this: - Modify mbuf_to_label() to extract the tag, returning NULL if not found. - Introduce mac_init_mbuf_tag() which does most of the work mac_init_mbuf() used to do, except on an m_tag rather than an mbuf. - Scale back mac_init_mbuf() to perform m_tag allocation and invoke mac_init_mbuf_tag(). - Replace mac_destroy_mbuf() with mac_destroy_mbuf_tag(), since m_tag's are now GC'd deep in the m_tag/mbuf code rather than at a higher level when mbufs are directly free()'d. - Add mac_copy_mbuf_tag() to support m_copy_pkthdr() and related notions. - Generally change all references to mbuf labels so that they use mbuf_to_label() rather than &mbuf->m_pkthdr.label. This required no changes in the MAC policies (yay!). - Tweak mbuf release routines to not call mac_destroy_mbuf(), tag destruction takes care of it for us now. - Remove MAC magic from m_copy_pkthdr() and m_move_pkthdr() -- the existing m_tag support does all this for us. Note that we can no longer just zero the m_tag list on the target mbuf, rather, we have to delete the chain because m_tag's will already be hung off freshly allocated mbuf's. - Tweak m_tag copying routines so that if we're copying a MAC m_tag, we don't do a binary copy, rather, we initialize the new storage and do a deep copy of the label. - Remove use of MAC_FLAG_INITIALIZED in a few bizarre places having to do with mbuf header copies previously. - When an mbuf is copied in ip_input(), we no longer need to explicitly copy the label because it will get handled by the m_tag code now. - No longer any weird handling of MAC labels in if_loop.c during header copies. - Add MPC_LOADTIME_FLAG_LABELMBUFS flag to Biba, MLS, mac_test. In mac_test, handle the label==NULL case, since it can be dynamically loaded. In order to improve performance with this change, introduce the notion of "lazy MAC label allocation" -- only allocate m_tag storage for MAC labels if we're running with a policy that uses MAC labels on mbufs. Policies declare this intent by setting the MPC_LOADTIME_FLAG_LABELMBUFS flag in their load-time flags field during declaration. Note: this opens up the possibility of post-boot policy modules getting back NULL slot entries even though they have policy invariants of non-NULL slot entries, as the policy might have been loaded after the mbuf was allocated, leaving the mbuf without label storage. Policies that cannot handle this case must be declared as NOTLATE, or must be modified. - mac_labelmbufs holds the current cumulative status as to whether any policies require mbuf labeling or not. This is updated whenever the active policy set changes by the function mac_policy_updateflags(). The function iterates the list and checks whether any have the flag set. Write access to this variable is protected by the policy list; read access is currently not protected for performance reasons. This might change if it causes problems. - Add MAC_POLICY_LIST_ASSERT_EXCLUSIVE() to permit the flags update function to assert appropriate locks. - This makes allocation in mac_init_mbuf() conditional on the flag. Reviewed by: sam Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2003-04-14 20:39:06 +00:00
/* Note: with MAC, this may not be a good assertion. */
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
KASSERT(SLIST_EMPTY(&to->m_pkthdr.tags), ("m_dup_pkthdr: to has tags"));
#endif
Make sure we don't call mbuf allocation functions with mutexes held. Discussed with: rwatson
2004-07-21 07:12:24 +00:00
MBUF_CHECKSLEEP(how);
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
#ifdef MAC
if (to->m_flags & M_PKTHDR)
Move MAC label storage for mbufs into m_tags from the m_pkthdr structure, returning some additional room in the first mbuf in a chain, and avoiding feature-specific contents in the mbuf header. To do this: - Modify mbuf_to_label() to extract the tag, returning NULL if not found. - Introduce mac_init_mbuf_tag() which does most of the work mac_init_mbuf() used to do, except on an m_tag rather than an mbuf. - Scale back mac_init_mbuf() to perform m_tag allocation and invoke mac_init_mbuf_tag(). - Replace mac_destroy_mbuf() with mac_destroy_mbuf_tag(), since m_tag's are now GC'd deep in the m_tag/mbuf code rather than at a higher level when mbufs are directly free()'d. - Add mac_copy_mbuf_tag() to support m_copy_pkthdr() and related notions. - Generally change all references to mbuf labels so that they use mbuf_to_label() rather than &mbuf->m_pkthdr.label. This required no changes in the MAC policies (yay!). - Tweak mbuf release routines to not call mac_destroy_mbuf(), tag destruction takes care of it for us now. - Remove MAC magic from m_copy_pkthdr() and m_move_pkthdr() -- the existing m_tag support does all this for us. Note that we can no longer just zero the m_tag list on the target mbuf, rather, we have to delete the chain because m_tag's will already be hung off freshly allocated mbuf's. - Tweak m_tag copying routines so that if we're copying a MAC m_tag, we don't do a binary copy, rather, we initialize the new storage and do a deep copy of the label. - Remove use of MAC_FLAG_INITIALIZED in a few bizarre places having to do with mbuf header copies previously. - When an mbuf is copied in ip_input(), we no longer need to explicitly copy the label because it will get handled by the m_tag code now. - No longer any weird handling of MAC labels in if_loop.c during header copies. - Add MPC_LOADTIME_FLAG_LABELMBUFS flag to Biba, MLS, mac_test. In mac_test, handle the label==NULL case, since it can be dynamically loaded. In order to improve performance with this change, introduce the notion of "lazy MAC label allocation" -- only allocate m_tag storage for MAC labels if we're running with a policy that uses MAC labels on mbufs. Policies declare this intent by setting the MPC_LOADTIME_FLAG_LABELMBUFS flag in their load-time flags field during declaration. Note: this opens up the possibility of post-boot policy modules getting back NULL slot entries even though they have policy invariants of non-NULL slot entries, as the policy might have been loaded after the mbuf was allocated, leaving the mbuf without label storage. Policies that cannot handle this case must be declared as NOTLATE, or must be modified. - mac_labelmbufs holds the current cumulative status as to whether any policies require mbuf labeling or not. This is updated whenever the active policy set changes by the function mac_policy_updateflags(). The function iterates the list and checks whether any have the flag set. Write access to this variable is protected by the policy list; read access is currently not protected for performance reasons. This might change if it causes problems. - Add MAC_POLICY_LIST_ASSERT_EXCLUSIVE() to permit the flags update function to assert appropriate locks. - This makes allocation in mac_init_mbuf() conditional on the flag. Reviewed by: sam Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2003-04-14 20:39:06 +00:00
m_tag_delete_chain(to, NULL);
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
#endif
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
to->m_flags = (from->m_flags & M_COPYFLAGS) |
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
(to->m_flags & (M_EXT | M_EXTPG));
Allow m_dup_pkthdr to accept mbufs with attached clusters as targets. Submitted by: bmilekic
2003-03-28 05:57:48 +00:00
if ((to->m_flags & M_EXT) == 0)
to->m_data = to->m_pktdat;
Make M_COPY_PKTHDR() macro into a wrapper for a m_copy_pkthdr() function. This permits conditionally compiled extensions to the packet header copying semantic, such as extensions to copy MAC labels. Reviewed by: bmilekic Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
2002-07-30 18:28:58 +00:00
to->m_pkthdr = from->m_pkthdr;
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
if (from->m_pkthdr.csum_flags & CSUM_SND_TAG)
m_snd_tag_ref(from->m_pkthdr.snd_tag);
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
SLIST_INIT(&to->m_pkthdr.tags);
Remove historical macro. Sponsored by: Nginx, Inc.
2014-01-16 13:42:50 +00:00
return (m_tag_copy_chain(to, from, how));
Make M_COPY_PKTHDR() macro into a wrapper for a m_copy_pkthdr() function. This permits conditionally compiled extensions to the packet header copying semantic, such as extensions to copy MAC labels. Reviewed by: bmilekic Obtained from: TrustedBSD Project Sponsored by: DARPA, NAI Labs
2002-07-30 18:28:58 +00:00
}
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Lesser-used path for M_PREPEND:
* allocate new mbuf to prepend to chain,
* copy junk along.
*/
struct mbuf *
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
m_prepend(struct mbuf *m, int len, int how)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
struct mbuf *mn;
Three fixes: - Make m_prepend use m_gethdr instead of m_get where appropriate - Make m_copym use m_gethdr instead of m_get where appropriate - Add a call to m_fixhdr in m_defrag; m_defrag can't deal with corrupted pkthdr.len counts. MFC after: 3 days
2003-07-19 06:03:48 +00:00
if (m->m_flags & M_PKTHDR)
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
mn = m_gethdr(how, m->m_type);
Three fixes: - Make m_prepend use m_gethdr instead of m_get where appropriate - Make m_copym use m_gethdr instead of m_get where appropriate - Add a call to m_fixhdr in m_defrag; m_defrag can't deal with corrupted pkthdr.len counts. MFC after: 3 days
2003-07-19 06:03:48 +00:00
else
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
mn = m_get(how, m->m_type);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (mn == NULL) {
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
m_freem(m);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
return (NULL);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
Move MAC label storage for mbufs into m_tags from the m_pkthdr structure, returning some additional room in the first mbuf in a chain, and avoiding feature-specific contents in the mbuf header. To do this: - Modify mbuf_to_label() to extract the tag, returning NULL if not found. - Introduce mac_init_mbuf_tag() which does most of the work mac_init_mbuf() used to do, except on an m_tag rather than an mbuf. - Scale back mac_init_mbuf() to perform m_tag allocation and invoke mac_init_mbuf_tag(). - Replace mac_destroy_mbuf() with mac_destroy_mbuf_tag(), since m_tag's are now GC'd deep in the m_tag/mbuf code rather than at a higher level when mbufs are directly free()'d. - Add mac_copy_mbuf_tag() to support m_copy_pkthdr() and related notions. - Generally change all references to mbuf labels so that they use mbuf_to_label() rather than &mbuf->m_pkthdr.label. This required no changes in the MAC policies (yay!). - Tweak mbuf release routines to not call mac_destroy_mbuf(), tag destruction takes care of it for us now. - Remove MAC magic from m_copy_pkthdr() and m_move_pkthdr() -- the existing m_tag support does all this for us. Note that we can no longer just zero the m_tag list on the target mbuf, rather, we have to delete the chain because m_tag's will already be hung off freshly allocated mbuf's. - Tweak m_tag copying routines so that if we're copying a MAC m_tag, we don't do a binary copy, rather, we initialize the new storage and do a deep copy of the label. - Remove use of MAC_FLAG_INITIALIZED in a few bizarre places having to do with mbuf header copies previously. - When an mbuf is copied in ip_input(), we no longer need to explicitly copy the label because it will get handled by the m_tag code now. - No longer any weird handling of MAC labels in if_loop.c during header copies. - Add MPC_LOADTIME_FLAG_LABELMBUFS flag to Biba, MLS, mac_test. In mac_test, handle the label==NULL case, since it can be dynamically loaded. In order to improve performance with this change, introduce the notion of "lazy MAC label allocation" -- only allocate m_tag storage for MAC labels if we're running with a policy that uses MAC labels on mbufs. Policies declare this intent by setting the MPC_LOADTIME_FLAG_LABELMBUFS flag in their load-time flags field during declaration. Note: this opens up the possibility of post-boot policy modules getting back NULL slot entries even though they have policy invariants of non-NULL slot entries, as the policy might have been loaded after the mbuf was allocated, leaving the mbuf without label storage. Policies that cannot handle this case must be declared as NOTLATE, or must be modified. - mac_labelmbufs holds the current cumulative status as to whether any policies require mbuf labeling or not. This is updated whenever the active policy set changes by the function mac_policy_updateflags(). The function iterates the list and checks whether any have the flag set. Write access to this variable is protected by the policy list; read access is currently not protected for performance reasons. This might change if it causes problems. - Add MAC_POLICY_LIST_ASSERT_EXCLUSIVE() to permit the flags update function to assert appropriate locks. - This makes allocation in mac_init_mbuf() conditional on the flag. Reviewed by: sam Obtained from: TrustedBSD Project Sponsored by: DARPA, Network Associates Laboratories
2003-04-14 20:39:06 +00:00
if (m->m_flags & M_PKTHDR)
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
m_move_pkthdr(mn, m);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
mn->m_next = m;
m = mn;
To ease changes to underlying mbuf structure and the mbuf allocator, reduce the knowledge of mbuf layout, and in particular constants such as M_EXT, MLEN, MHLEN, and so on, in mbuf consumers by unifying various alignment utility functions (M_ALIGN(), MH_ALIGN(), MEXT_ALIGN() in a single M_ALIGN() macro, implemented by a now-inlined m_align() function: - Move m_align() from uipc_mbuf.c to mbuf.h; mark as __inline. - Reimplement M_ALIGN(), MH_ALIGN(), and MEXT_ALIGN() using m_align(). - Update consumers around the tree to simply use M_ALIGN(). This change eliminates a number of cases where mbuf consumers must be aware of whether or not mbufs returned by the allocator use external storage, but also assumptions about the size of the returned mbuf. This will make it easier to introduce changes in how we use external storage, as well as features such as variable-size mbufs. Differential Revision: https://reviews.freebsd.org/D1436 Reviewed by: glebius, trasz, gnn, bz Sponsored by: EMC / Isilon Storage Division
2015-01-05 09:58:32 +00:00
if (len < M_SIZE(m))
M_ALIGN(m, len);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
m->m_len = len;
return (m);
}
/*
* Make a copy of an mbuf chain starting "off0" bytes from the beginning,
* continuing for "len" bytes. If len is M_COPYALL, copy to end of mbuf.
Mechanically substitute flags from historic mbuf allocator with malloc(9) flags within sys. Exceptions: - sys/contrib not touched - sys/mbuf.h edited manually
2012-12-05 08:04:20 +00:00
* The wait parameter is a choice of M_WAITOK/M_NOWAIT from caller.
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
* Note that the copy is read-only, because clusters are not copied,
* only their reference counts are incremented.
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*/
struct mbuf *
New way to manage reference counting of mbuf external storage. The m_ext.ext_cnt pointer becomes a union. It can now hold the refcount value itself. To tell that m_ext.ext_flags flag EXT_FLAG_EMBREF is used. The first mbuf to attach a cluster stores the refcount. The further mbufs to reference the cluster point at refcount in the first mbuf. The first mbuf is freed only when the last reference is freed. The benefit over refcounts stored in separate slabs is that now refcounts of different, unrelated mbufs do not share a cache line. For EXT_EXTREF mbufs the zone_ext_refcnt is no longer needed, and m_extadd() becomes void, making widely used M_EXTADD macro safe. For EXT_SFBUF mbufs the sf_ext_ref() is removed, which was an optimization exactly against the cache aliasing problem with regular refcounting. Discussed with: rrs, rwatson, gnn, hiren, sbruno, np Reviewed by: rrs Differential Revision: https://reviews.freebsd.org/D5396 Sponsored by: Netflix
2016-03-01 00:17:14 +00:00
m_copym(struct mbuf *m, int off0, int len, int wait)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
struct mbuf *n, **np;
int off = off0;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
struct mbuf *top;
int copyhdr = 0;
change identical and "programming error" panic("mcopy*")'s into more verbose messages using KASSERT. Reviewed by: eivind, des
1999-10-13 09:55:42 +00:00
KASSERT(off >= 0, ("m_copym, negative off %d", off));
KASSERT(len >= 0, ("m_copym, negative len %d", len));
Make sure we don't call mbuf allocation functions with mutexes held. Discussed with: rwatson
2004-07-21 07:12:24 +00:00
MBUF_CHECKSLEEP(wait);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (off == 0 && m->m_flags & M_PKTHDR)
copyhdr = 1;
while (off > 0) {
change identical and "programming error" panic("mcopy*")'s into more verbose messages using KASSERT. Reviewed by: eivind, des
1999-10-13 09:55:42 +00:00
KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (off < m->m_len)
break;
off -= m->m_len;
m = m->m_next;
}
np = &top;
kern: for pointers replace 0 with NULL. These are mostly cosmetical, no functional change. Found with devel/coccinelle.
2016-04-15 16:10:11 +00:00
top = NULL;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
while (len > 0) {
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (m == NULL) {
Whitespace only.
2014-05-30 08:22:58 +00:00
KASSERT(len == M_COPYALL,
change identical and "programming error" panic("mcopy*")'s into more verbose messages using KASSERT. Reviewed by: eivind, des
1999-10-13 09:55:42 +00:00
("m_copym, length > size of mbuf chain"));
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
break;
}
Three fixes: - Make m_prepend use m_gethdr instead of m_get where appropriate - Make m_copym use m_gethdr instead of m_get where appropriate - Add a call to m_fixhdr in m_defrag; m_defrag can't deal with corrupted pkthdr.len counts. MFC after: 3 days
2003-07-19 06:03:48 +00:00
if (copyhdr)
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
n = m_gethdr(wait, m->m_type);
Three fixes: - Make m_prepend use m_gethdr instead of m_get where appropriate - Make m_copym use m_gethdr instead of m_get where appropriate - Add a call to m_fixhdr in m_defrag; m_defrag can't deal with corrupted pkthdr.len counts. MFC after: 3 days
2003-07-19 06:03:48 +00:00
else
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
n = m_get(wait, m->m_type);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*np = n;
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (n == NULL)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
goto nospace;
if (copyhdr) {
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
if (!m_dup_pkthdr(n, m, wait))
goto nospace;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (len == M_COPYALL)
n->m_pkthdr.len -= off0;
else
n->m_pkthdr.len = len;
copyhdr = 0;
}
n->m_len = min(len, m->m_len - off);
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
if (m->m_flags & (M_EXT|M_EXTPG)) {
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
n->m_data = m->m_data + off;
Mandatory mbuf cluster reference counting and groundwork for UMA based jumbo 9k and jumbo 16k cluster support. All mbuf's with external storage attached are mandatory reference counted. For clusters and jumbo clusters UMA provides the refcnt storage directly. It does not have to be separatly allocated. Any other type of external storage gets its own refcnt allocated from an UMA mbuf refcnt zone instead of normal kernel malloc. The refcount API MEXT_ADD_REF() and MEXT_REM_REF() is no longer publically accessible. The proper m_* functions have to be used. mb_ctor_clust() and mb_dtor_clust() both handle normal 2K as well as 9k and 16k clusters. Clusters and jumbo clusters may be obtained without attaching it immideatly to an mbuf. This is for high performance cluster allocation in network drivers where mbufs are attached after the cluster has been filled. Tested by: rwatson Sponsored by: TCP/IP Optimizations Fundraise 2005
2005-11-02 16:20:36 +00:00
mb_dupcl(n, m);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
} else
bcopy(mtod(m, caddr_t)+off, mtod(n, caddr_t),
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
(u_int)n->m_len);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (len != M_COPYALL)
len -= n->m_len;
off = 0;
m = m->m_next;
np = &n->m_next;
}
Introduce numerous SMP friendly changes to the mbuf allocator. Namely, introduce a modified allocation mechanism for mbufs and mbuf clusters; one which can scale under SMP and which offers the possibility of resource reclamation to be implemented in the future. Notable advantages: o Reduce contention for SMP by offering per-CPU pools and locks. o Better use of data cache due to per-CPU pools. o Much less code cache pollution due to excessively large allocation macros. o Framework for `grouping' objects from same page together so as to be able to possibly free wired-down pages back to the system if they are no longer needed by the network stacks. Additional things changed with this addition: - Moved some mbuf specific declarations and initializations from sys/conf/param.c into mbuf-specific code where they belong. - m_getclr() has been renamed to m_get_clrd() because the old name is really confusing. m_getclr() HAS been preserved though and is defined to the new name. No tree sweep has been done "to change the interface," as the old name will continue to be supported and is not depracated. The change was merely done because m_getclr() sounds too much like "m_get a cluster." - TEMPORARILY disabled mbtypes statistics displaying in netstat(1) and systat(1) (see TODO below). - Fixed systat(1) to display number of "free mbufs" based on new per-CPU stat structures. - Fixed netstat(1) to display new per-CPU stats based on sysctl-exported per-CPU stat structures. All infos are fetched via sysctl. TODO (in order of priority): - Re-enable mbtypes statistics in both netstat(1) and systat(1) after introducing an SMP friendly way to collect the mbtypes stats under the already introduced per-CPU locks (i.e. hopefully don't use atomic() - it seems too costly for a mere stat update, especially when other locks are already present). - Optionally have systat(1) display not only "total free mbufs" but also "total free mbufs per CPU pool." - Fix minor length-fetching issues in netstat(1) related to recently re-enabled option to read mbuf stats from a core file. - Move reference counters at least for mbuf clusters into an unused portion of the cluster itself, to save space and need to allocate a counter. - Look into introducing resource freeing possibly from a kproc. Reviewed by (in parts): jlemon, jake, silby, terry Tested by: jlemon (Intel & Alpha), mjacob (Intel & Alpha) Preliminary performance measurements: jlemon (and me, obviously) URL: http://people.freebsd.org/~bmilekic/mb_alloc/
2001-06-22 06:35:32 +00:00
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
return (top);
nospace:
m_freem(top);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
return (NULL);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
/*
* Copy an entire packet, including header (which must be present).
* An optimization of the common case `m_copym(m, 0, M_COPYALL, how)'.
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
* Note that the copy is read-only, because clusters are not copied,
* only their reference counts are incremented.
Preserve alignment of first mbuf in m_copypacket. This is useful when doing copies of packet where some leading space has been preallocated to insert protocol headers. Note that there are in fact almost no users of m_copypacket. MFC candidate.
2001-02-20 08:23:41 +00:00
* Preserve alignment of the first mbuf so if the creator has left
* some room at the beginning (e.g. for inserting protocol headers)
* the copies still have the room available.
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
*/
struct mbuf *
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
m_copypacket(struct mbuf *m, int how)
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
{
struct mbuf *top, *n, *o;
Make sure we don't call mbuf allocation functions with mutexes held. Discussed with: rwatson
2004-07-21 07:12:24 +00:00
MBUF_CHECKSLEEP(how);
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
n = m_get(how, m->m_type);
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
top = n;
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (n == NULL)
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
goto nospace;
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
if (!m_dup_pkthdr(n, m, how))
goto nospace;
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
n->m_len = m->m_len;
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
if (m->m_flags & (M_EXT|M_EXTPG)) {
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
n->m_data = m->m_data;
Mandatory mbuf cluster reference counting and groundwork for UMA based jumbo 9k and jumbo 16k cluster support. All mbuf's with external storage attached are mandatory reference counted. For clusters and jumbo clusters UMA provides the refcnt storage directly. It does not have to be separatly allocated. Any other type of external storage gets its own refcnt allocated from an UMA mbuf refcnt zone instead of normal kernel malloc. The refcount API MEXT_ADD_REF() and MEXT_REM_REF() is no longer publically accessible. The proper m_* functions have to be used. mb_ctor_clust() and mb_dtor_clust() both handle normal 2K as well as 9k and 16k clusters. Clusters and jumbo clusters may be obtained without attaching it immideatly to an mbuf. This is for high performance cluster allocation in network drivers where mbufs are attached after the cluster has been filled. Tested by: rwatson Sponsored by: TCP/IP Optimizations Fundraise 2005
2005-11-02 16:20:36 +00:00
mb_dupcl(n, m);
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
} else {
Preserve alignment of first mbuf in m_copypacket. This is useful when doing copies of packet where some leading space has been preallocated to insert protocol headers. Note that there are in fact almost no users of m_copypacket. MFC candidate.
2001-02-20 08:23:41 +00:00
n->m_data = n->m_pktdat + (m->m_data - m->m_pktdat );
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
bcopy(mtod(m, char *), mtod(n, char *), n->m_len);
}
m = m->m_next;
while (m) {
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
o = m_get(how, m->m_type);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (o == NULL)
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
goto nospace;
n->m_next = o;
n = n->m_next;
n->m_len = m->m_len;
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
if (m->m_flags & (M_EXT|M_EXTPG)) {
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
n->m_data = m->m_data;
Mandatory mbuf cluster reference counting and groundwork for UMA based jumbo 9k and jumbo 16k cluster support. All mbuf's with external storage attached are mandatory reference counted. For clusters and jumbo clusters UMA provides the refcnt storage directly. It does not have to be separatly allocated. Any other type of external storage gets its own refcnt allocated from an UMA mbuf refcnt zone instead of normal kernel malloc. The refcount API MEXT_ADD_REF() and MEXT_REM_REF() is no longer publically accessible. The proper m_* functions have to be used. mb_ctor_clust() and mb_dtor_clust() both handle normal 2K as well as 9k and 16k clusters. Clusters and jumbo clusters may be obtained without attaching it immideatly to an mbuf. This is for high performance cluster allocation in network drivers where mbufs are attached after the cluster has been filled. Tested by: rwatson Sponsored by: TCP/IP Optimizations Fundraise 2005
2005-11-02 16:20:36 +00:00
mb_dupcl(n, m);
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
} else {
bcopy(mtod(m, char *), mtod(n, char *), n->m_len);
}
m = m->m_next;
}
return top;
nospace:
m_freem(top);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
return (NULL);
Our new-old mbugf allocator. This is actually something of a blast from the past, since it returns to the old system of allocating mbufs out of a private area rather than using the kernel malloc(). While this may seem like a backwards step to some, the new allocator is some 20% faster than the old one and has much better caching properties. Written by: John Wroclawski <jtw@lcs.mit.edu>
1996-05-08 19:38:27 +00:00
}
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
static void
m_copyfromunmapped(const struct mbuf *m, int off, int len, caddr_t cp)
{
struct iovec iov;
struct uio uio;
int error;
KASSERT(off >= 0, ("m_copyfromunmapped: negative off %d", off));
KASSERT(len >= 0, ("m_copyfromunmapped: negative len %d", len));
KASSERT(off < m->m_len,
("m_copyfromunmapped: len exceeds mbuf length"));
iov.iov_base = cp;
iov.iov_len = len;
uio.uio_resid = len;
uio.uio_iov = &iov;
uio.uio_segflg = UIO_SYSSPACE;
uio.uio_iovcnt = 1;
uio.uio_offset = 0;
uio.uio_rw = UIO_READ;
error = m_unmappedtouio(m, off, &uio, len);
KASSERT(error == 0, ("m_unmappedtouio failed: off %d, len %d", off,
len));
}
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Copy data from an mbuf chain starting "off" bytes from the beginning,
* continuing for "len" bytes, into the indicated buffer.
*/
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
void
Forgot to remove this un-needed test. (M_WAITOK won't fail) I vaguely remember someone once proving it COULD return NULL.. was that changed? Reminded by: BDE MFC after: 2 weeks
2001-08-19 04:30:13 +00:00
m_copydata(const struct mbuf *m, int off, int len, caddr_t cp)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
u_int count;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
change identical and "programming error" panic("mcopy*")'s into more verbose messages using KASSERT. Reviewed by: eivind, des
1999-10-13 09:55:42 +00:00
KASSERT(off >= 0, ("m_copydata, negative off %d", off));
KASSERT(len >= 0, ("m_copydata, negative len %d", len));
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
while (off > 0) {
change identical and "programming error" panic("mcopy*")'s into more verbose messages using KASSERT. Reviewed by: eivind, des
1999-10-13 09:55:42 +00:00
KASSERT(m != NULL, ("m_copydata, offset > size of mbuf chain"));
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (off < m->m_len)
break;
off -= m->m_len;
m = m->m_next;
}
while (len > 0) {
change identical and "programming error" panic("mcopy*")'s into more verbose messages using KASSERT. Reviewed by: eivind, des
1999-10-13 09:55:42 +00:00
KASSERT(m != NULL, ("m_copydata, length > size of mbuf chain"));
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
count = min(m->m_len - off, len);
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
if ((m->m_flags & M_EXTPG) != 0)
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
m_copyfromunmapped(m, off, count, cp);
else
bcopy(mtod(m, caddr_t) + off, cp, count);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
len -= count;
cp += count;
off = 0;
m = m->m_next;
}
}
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
/*
* Copy a packet header mbuf chain into a completely new chain, including
* copying any mbuf clusters. Use this instead of m_copypacket() when
* you need a writable copy of an mbuf chain.
*/
struct mbuf *
Add const-qualifiers for source mbuf argument in m_dup(), m_copym(), m_dup_pkthdr() and m_tag_copy_chain().
2015-08-08 15:50:46 +00:00
m_dup(const struct mbuf *m, int how)
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
{
struct mbuf **p, *top = NULL;
int remain, moff, nsize;
Make sure we don't call mbuf allocation functions with mutexes held. Discussed with: rwatson
2004-07-21 07:12:24 +00:00
MBUF_CHECKSLEEP(how);
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
/* Sanity check */
if (m == NULL)
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
return (NULL);
Introduce an M_ASSERTPKTHDR() macro which performs the very common task of asserting that an mbuf has a packet header. Use it instead of hand- rolled versions wherever applicable. Submitted by: Hiten Pandya <hiten@unixdaemons.com>
2003-04-08 14:25:47 +00:00
M_ASSERTPKTHDR(m);
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
/* While there's more data, get a new mbuf, tack it on, and fill it */
remain = m->m_pkthdr.len;
moff = 0;
p = &top;
while (remain > 0 || top == NULL) { /* allow m->m_pkthdr.len == 0 */
struct mbuf *n;
/* Get the next new mbuf */
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
if (remain >= MINCLSIZE) {
n = m_getcl(how, m->m_type, 0);
nsize = MCLBYTES;
} else {
n = m_get(how, m->m_type);
nsize = MLEN;
}
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
if (n == NULL)
goto nospace;
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
if (top == NULL) { /* First one, must be PKTHDR */
if (!m_dup_pkthdr(n, m, how)) {
m_free(n);
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
goto nospace;
}
When using m_dup(9) to copy more than MHLEN bytes of data, don't create an mbuf chain that starts with a cluster containing just MHLEN bytes. This happened because m_dup called m_get or m_getcl depending on the amount of data to copy, but then always set the size available in the first mbuf to MHLEN. Submitted by: Matt Koivisto <mkoivisto at sandvine dot com> Approved by: jmg Silence from: rwatson (mentor)
2005-12-14 23:34:26 +00:00
if ((n->m_flags & M_EXT) == 0)
nsize = MHLEN;
m_dup() is supposed to give a writable copy of an mbuf chain. It uses m_dup_pkthdr(), that uses M_COPYFLAGS mask to copy m_flags field. If original mbuf chain has M_RDONLY flag, its copy also will have it. Reset this flag explicitly. MFC after: 2 weeks
2015-05-07 18:35:01 +00:00
n->m_flags &= ~M_RDONLY;
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
}
n->m_len = 0;
/* Link it into the new chain */
*p = n;
p = &n->m_next;
/* Copy data from original mbuf(s) into new mbuf */
while (n->m_len < nsize && m != NULL) {
int chunk = min(nsize - n->m_len, m->m_len - moff);
bcopy(m->m_data + moff, n->m_data + n->m_len, chunk);
moff += chunk;
n->m_len += chunk;
remain -= chunk;
if (moff == m->m_len) {
m = m->m_next;
moff = 0;
}
}
/* Check correct total mbuf length */
KASSERT((remain > 0 && m != NULL) || (remain == 0 && m == NULL),
Update to C99, s/__FUNCTION__/__func__/.
2001-12-10 05:51:45 +00:00
("%s: bogus m_pkthdr.len", __func__));
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
}
return (top);
nospace:
m_freem(top);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
return (NULL);
The functions m_copym() and m_copypacket() return read-only copies, because in the case of mbuf clusters they only increment the reference count rather than actually copying the data. Add comments to this effect, and add a new routine called m_dup() that returns a real, writable copy of an mbuf chain. This is preliminary work required for implementing 'ipfw tee'. Reviewed by: julian
1999-12-01 22:31:32 +00:00
}
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Concatenate mbuf chain n to m.
* Both chains must be of the same type (e.g. MT_DATA).
* Any m_pkthdr is not updated.
*/
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
void
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
m_cat(struct mbuf *m, struct mbuf *n)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
while (m->m_next)
m = m->m_next;
while (n) {
Improve m_cat() by being able to also merge contents from M_EXT mbuf's by doing proper testing with M_WRITABLE(). In m_collapse() replace an incomplete manual check for M_RDONLY with the M_WRITABLE() macro that also tests for shared buffers and other cases that make a particular mbuf immutable. MFC after: 2 weeks
2012-10-28 18:38:51 +00:00
if (!M_WRITABLE(m) ||
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
(n->m_flags & M_EXTPG) != 0 ||
Improve m_cat() by being able to also merge contents from M_EXT mbuf's by doing proper testing with M_WRITABLE(). In m_collapse() replace an incomplete manual check for M_RDONLY with the M_WRITABLE() macro that also tests for shared buffers and other cases that make a particular mbuf immutable. MFC after: 2 weeks
2012-10-28 18:38:51 +00:00
M_TRAILINGSPACE(m) < n->m_len) {
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/* just join the two chains */
m->m_next = n;
return;
}
/* splat the data from one into the other */
bcopy(mtod(n, caddr_t), mtod(m, caddr_t) + m->m_len,
(u_int)n->m_len);
m->m_len += n->m_len;
n = m_free(n);
}
}
Provide m_catpkt(), a wrapper around m_cat() that deals with M_PKTHDR mbufs. Sponsored by: Netflix Sponsored by: Nginx, Inc.
2014-09-04 09:07:14 +00:00
/*
* Concatenate two pkthdr mbuf chains.
*/
void
m_catpkt(struct mbuf *m, struct mbuf *n)
{
M_ASSERTPKTHDR(m);
M_ASSERTPKTHDR(n);
m->m_pkthdr.len += n->m_pkthdr.len;
Merge from projects/sendfile: extend protocols API to support sending not ready data: o Add new flag to pru_send() flags - PRUS_NOTREADY. o Add new protocol method pru_ready(). Sponsored by: Nginx, Inc. Sponsored by: Netflix
2014-11-30 13:24:21 +00:00
m_demote(n, 1, 0);
Provide m_catpkt(), a wrapper around m_cat() that deals with M_PKTHDR mbufs. Sponsored by: Netflix Sponsored by: Nginx, Inc.
2014-09-04 09:07:14 +00:00
m_cat(m, n);
}
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
void
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
m_adj(struct mbuf *mp, int req_len)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
int len = req_len;
struct mbuf *m;
int count;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if ((m = mp) == NULL)
return;
if (len >= 0) {
/*
* Trim from head.
*/
while (m != NULL && len > 0) {
if (m->m_len <= len) {
len -= m->m_len;
m->m_len = 0;
m = m->m_next;
} else {
m->m_len -= len;
m->m_data += len;
len = 0;
}
}
if (mp->m_flags & M_PKTHDR)
Remove a redundant variable assignment. Reviewed by: bz, rwatson MFC after: 3 days
2010-05-01 18:34:50 +00:00
mp->m_pkthdr.len -= (req_len - len);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
} else {
/*
* Trim from tail. Scan the mbuf chain,
* calculating its length and finding the last mbuf.
* If the adjustment only affects this mbuf, then just
* adjust and return. Otherwise, rescan and truncate
* after the remaining size.
*/
len = -len;
count = 0;
for (;;) {
count += m->m_len;
if (m->m_next == (struct mbuf *)0)
break;
m = m->m_next;
}
if (m->m_len >= len) {
m->m_len -= len;
if (mp->m_flags & M_PKTHDR)
mp->m_pkthdr.len -= len;
return;
}
count -= len;
if (count < 0)
count = 0;
/*
* Correct length for chain is "count".
* Find the mbuf with last data, adjust its length,
* and toss data from remaining mbufs on chain.
*/
m = mp;
if (m->m_flags & M_PKTHDR)
m->m_pkthdr.len = count;
for (; m; m = m->m_next) {
if (m->m_len >= count) {
m->m_len = count;
change m_adj to reclaim unused mbufs instead of zero'ing m_len when trim'ing space off the back of a chain; this is indirect solution to a potential null ptr deref Noticed by: Coverity Prevent analysis tool (null ptr deref) Reviewed by: dg, rwatson
2005-02-24 00:40:33 +00:00
if (m->m_next != NULL) {
m_freem(m->m_next);
m->m_next = NULL;
}
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
break;
}
count -= m->m_len;
}
}
}
/*
* Rearange an mbuf chain so that len bytes are contiguous
Since the macro dtom() has been removed, fix comments about the dtom. Reviewed by: glebius
2012-10-29 10:04:28 +00:00
* and in the data area of an mbuf (so that mtod will work
* for a structure of size len). Returns the resulting
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
* mbuf chain on success, frees it and returns null on failure.
* If there is room, it will add up to max_protohdr-len extra bytes to the
* contiguous region in an attempt to avoid being called next time.
*/
struct mbuf *
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
m_pullup(struct mbuf *n, int len)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
struct mbuf *m;
int count;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
int space;
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
KASSERT((n->m_flags & M_EXTPG) == 0,
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
("%s: unmapped mbuf %p", __func__, n));
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* If first mbuf has no cluster, and has room for len bytes
* without shifting current data, pullup into it,
* otherwise allocate a new mbuf to prepend to the chain.
*/
if ((n->m_flags & M_EXT) == 0 &&
n->m_data + len < &n->m_dat[MLEN] && n->m_next) {
if (n->m_len >= len)
return (n);
m = n;
n = n->m_next;
len -= m->m_len;
} else {
if (len > MHLEN)
goto bad;
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
m = m_get(M_NOWAIT, n->m_type);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (m == NULL)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
goto bad;
Correct mbuf packet header propagation. Previously, packet headers were sometimes propagated using M_COPY_PKTHDR which actually did something between a "move" and a "copy" operation. This is replaced by M_MOVE_PKTHDR (which copies the pkthdr contents and "removes" it from the source mbuf) and m_dup_pkthdr which copies the packet header contents including any m_tag chain. This corrects numerous problems whereby mbuf tags could be lost during packet manipulations. These changes also introduce arguments to m_tag_copy and m_tag_copy_chain to specify if the tag copy work should potentially block. This introduces an incompatibility with openbsd which we may want to revisit. Note that move/dup of packet headers does not handle target mbufs that have a cluster bound to them. We may want to support this; for now we watch for it with an assert. Finally, M_COPYFLAGS was updated to include M_FIRSTFRAG|M_LASTFRAG. Supported by: Vernier Networks Reviewed by: Robert Watson <rwatson@FreeBSD.org>
2002-12-30 20:22:40 +00:00
if (n->m_flags & M_PKTHDR)
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
m_move_pkthdr(m, n);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
space = &m->m_dat[MLEN] - (m->m_data + m->m_len);
do {
count = min(min(max(len, max_protohdr), space), n->m_len);
bcopy(mtod(n, caddr_t), mtod(m, caddr_t) + m->m_len,
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
(u_int)count);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
len -= count;
m->m_len += count;
n->m_len -= count;
space -= count;
if (n->m_len)
n->m_data += count;
else
n = m_free(n);
} while (len > 0 && n);
if (len > 0) {
(void) m_free(m);
goto bad;
}
m->m_next = n;
return (m);
bad:
m_freem(n);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
return (NULL);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
add m_copyup function.. This can be used to help make our ip stack less alignment restrictive, and help performance on some ethernet cards which currently copy the entire packet a couple bytes to get the packet aligned properly... Wordsmithing by: dwhite Obtained from: NetBSD (code only) I'll clean it up later: rwatson
2005-03-17 19:34:57 +00:00
/*
* Like m_pullup(), except a new mbuf is always allocated, and we allow
* the amount of empty space before the data in the new mbuf to be specified
* (in the event that the caller expects to prepend later).
*/
struct mbuf *
m_copyup(struct mbuf *n, int len, int dstoff)
{
struct mbuf *m;
int count, space;
if (len > (MHLEN - dstoff))
goto bad;
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
m = m_get(M_NOWAIT, n->m_type);
add m_copyup function.. This can be used to help make our ip stack less alignment restrictive, and help performance on some ethernet cards which currently copy the entire packet a couple bytes to get the packet aligned properly... Wordsmithing by: dwhite Obtained from: NetBSD (code only) I'll clean it up later: rwatson
2005-03-17 19:34:57 +00:00
if (m == NULL)
goto bad;
if (n->m_flags & M_PKTHDR)
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
m_move_pkthdr(m, n);
add m_copyup function.. This can be used to help make our ip stack less alignment restrictive, and help performance on some ethernet cards which currently copy the entire packet a couple bytes to get the packet aligned properly... Wordsmithing by: dwhite Obtained from: NetBSD (code only) I'll clean it up later: rwatson
2005-03-17 19:34:57 +00:00
m->m_data += dstoff;
space = &m->m_dat[MLEN] - (m->m_data + m->m_len);
do {
count = min(min(max(len, max_protohdr), space), n->m_len);
memcpy(mtod(m, caddr_t) + m->m_len, mtod(n, caddr_t),
(unsigned)count);
len -= count;
m->m_len += count;
n->m_len -= count;
space -= count;
if (n->m_len)
n->m_data += count;
else
n = m_free(n);
} while (len > 0 && n);
if (len > 0) {
(void) m_free(m);
goto bad;
}
m->m_next = n;
return (m);
bad:
m_freem(n);
return (NULL);
}
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
/*
* Partition an mbuf chain in two pieces, returning the tail --
* all but the first len0 bytes. In case of failure, it returns NULL and
* attempts to restore the chain to its original state.
Fix a bug in m_split(): the "m->m_ext.ext_size" field of an mbuf was being set to zero. This field indicates the total space in the external buffer and therefore should not be modified after the external buffer is added. Add a comment warning that the mbufs returned by m_split() might be read-only. Fix M_TRAILINGSPACE() to return zero if !M_WRITABLE(m). Reviewed by: freebsd-net Obtained from: Vernier Networks, Inc. MFC after: 1 week
2002-05-31 22:09:57 +00:00
*
* Note that the resulting mbufs might be read-only, because the new
* mbuf can end up sharing an mbuf cluster with the original mbuf if
* the "breaking point" happens to lie within a cluster mbuf. Use the
* M_WRITABLE() macro to check for this case.
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*/
struct mbuf *
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
m_split(struct mbuf *m0, int len0, int wait)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
struct mbuf *m, *n;
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
u_int len = len0, remain;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
Make sure we don't call mbuf allocation functions with mutexes held. Discussed with: rwatson
2004-07-21 07:12:24 +00:00
MBUF_CHECKSLEEP(wait);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
for (m = m0; m && len > m->m_len; m = m->m_next)
len -= m->m_len;
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (m == NULL)
return (NULL);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
remain = m->m_len - len;
Fix bug in m_split() in a case when split len matches len of the first mbuf, and the first mbuf is M_PKTHDR. PR: kern/176144 Submitted by: Jacques Fourie <jacques.fourie gmail.com>
2013-03-29 14:10:40 +00:00
if (m0->m_flags & M_PKTHDR && remain == 0) {
n = m_gethdr(wait, m0->m_type);
Fix a very bad typo from r248887. Submitted by: art
2013-11-14 09:45:33 +00:00
if (n == NULL)
Fix bug in m_split() in a case when split len matches len of the first mbuf, and the first mbuf is M_PKTHDR. PR: kern/176144 Submitted by: Jacques Fourie <jacques.fourie gmail.com>
2013-03-29 14:10:40 +00:00
return (NULL);
n->m_next = m->m_next;
m->m_next = NULL;
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
if (m0->m_pkthdr.csum_flags & CSUM_SND_TAG) {
n->m_pkthdr.snd_tag =
m_snd_tag_ref(m0->m_pkthdr.snd_tag);
n->m_pkthdr.csum_flags |= CSUM_SND_TAG;
} else
n->m_pkthdr.rcvif = m0->m_pkthdr.rcvif;
Fix bug in m_split() in a case when split len matches len of the first mbuf, and the first mbuf is M_PKTHDR. PR: kern/176144 Submitted by: Jacques Fourie <jacques.fourie gmail.com>
2013-03-29 14:10:40 +00:00
n->m_pkthdr.len = m0->m_pkthdr.len - len0;
m0->m_pkthdr.len = len0;
return (n);
} else if (m0->m_flags & M_PKTHDR) {
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
n = m_gethdr(wait, m0->m_type);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (n == NULL)
return (NULL);
Restructure mbuf send tags to provide stronger guarantees. - Perform ifp mismatch checks (to determine if a send tag is allocated for a different ifp than the one the packet is being output on), in ip_output() and ip6_output(). This avoids sending packets with send tags to ifnet drivers that don't support send tags. Since we are now checking for ifp mismatches before invoking if_output, we can now try to allocate a new tag before invoking if_output sending the original packet on the new tag if allocation succeeds. To avoid code duplication for the fragment and unfragmented cases, add ip_output_send() and ip6_output_send() as wrappers around if_output and nd6_output_ifp, respectively. All of the logic for setting send tags and dealing with send tag-related errors is done in these wrapper functions. For pseudo interfaces that wrap other network interfaces (vlan and lagg), wrapper send tags are now allocated so that ip*_output see the wrapper ifp as the ifp in the send tag. The if_transmit routines rewrite the send tags after performing an ifp mismatch check. If an ifp mismatch is detected, the transmit routines fail with EAGAIN. - To provide clearer life cycle management of send tags, especially in the presence of vlan and lagg wrapper tags, add a reference count to send tags managed via m_snd_tag_ref() and m_snd_tag_rele(). Provide a helper function (m_snd_tag_init()) for use by drivers supporting send tags. m_snd_tag_init() takes care of the if_ref on the ifp meaning that code alloating send tags via if_snd_tag_alloc no longer has to manage that manually. Similarly, m_snd_tag_rele drops the refcount on the ifp after invoking if_snd_tag_free when the last reference to a send tag is dropped. This also closes use after free races if there are pending packets in driver tx rings after the socket is closed (e.g. from tcpdrop). In order for m_free to work reliably, add a new CSUM_SND_TAG flag in csum_flags to indicate 'snd_tag' is set (rather than 'rcvif'). Drivers now also check this flag instead of checking snd_tag against NULL. This avoids false positive matches when a forwarded packet has a non-NULL rcvif that was treated as a send tag. - cxgbe was relying on snd_tag_free being called when the inp was detached so that it could kick the firmware to flush any pending work on the flow. This is because the driver doesn't require ACK messages from the firmware for every request, but instead does a kind of manual interrupt coalescing by only setting a flag to request a completion on a subset of requests. If all of the in-flight requests don't have the flag when the tag is detached from the inp, the flow might never return the credits. The current snd_tag_free command issues a flush command to force the credits to return. However, the credit return is what also frees the mbufs, and since those mbufs now hold references on the tag, this meant that snd_tag_free would never be called. To fix, explicitly drop the mbuf's reference on the snd tag when the mbuf is queued in the firmware work queue. This means that once the inp's reference on the tag goes away and all in-flight mbufs have been queued to the firmware, tag's refcount will drop to zero and snd_tag_free will kick in and send the flush request. Note that we need to avoid doing this in the middle of ethofld_tx(), so the driver grabs a temporary reference on the tag around that loop to defer the free to the end of the function in case it sends the last mbuf to the queue after the inp has dropped its reference on the tag. - mlx5 preallocates send tags and was using the ifp pointer even when the send tag wasn't in use. Explicitly use the ifp from other data structures instead. - Sprinkle some assertions in various places to assert that received packets don't have a send tag, and that other places that overwrite rcvif (e.g. 802.11 transmit) don't clobber a send tag pointer. Reviewed by: gallatin, hselasky, rgrimes, ae Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20117
2019-05-24 22:30:40 +00:00
if (m0->m_pkthdr.csum_flags & CSUM_SND_TAG) {
n->m_pkthdr.snd_tag =
m_snd_tag_ref(m0->m_pkthdr.snd_tag);
n->m_pkthdr.csum_flags |= CSUM_SND_TAG;
} else
n->m_pkthdr.rcvif = m0->m_pkthdr.rcvif;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
n->m_pkthdr.len = m0->m_pkthdr.len - len0;
m0->m_pkthdr.len = len0;
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
if (m->m_flags & (M_EXT|M_EXTPG))
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
goto extpacket;
if (remain > MHLEN) {
/* m can't be the lead packet */
To ease changes to underlying mbuf structure and the mbuf allocator, reduce the knowledge of mbuf layout, and in particular constants such as M_EXT, MLEN, MHLEN, and so on, in mbuf consumers by unifying various alignment utility functions (M_ALIGN(), MH_ALIGN(), MEXT_ALIGN() in a single M_ALIGN() macro, implemented by a now-inlined m_align() function: - Move m_align() from uipc_mbuf.c to mbuf.h; mark as __inline. - Reimplement M_ALIGN(), MH_ALIGN(), and MEXT_ALIGN() using m_align(). - Update consumers around the tree to simply use M_ALIGN(). This change eliminates a number of cases where mbuf consumers must be aware of whether or not mbufs returned by the allocator use external storage, but also assumptions about the size of the returned mbuf. This will make it easier to introduce changes in how we use external storage, as well as features such as variable-size mbufs. Differential Revision: https://reviews.freebsd.org/D1436 Reviewed by: glebius, trasz, gnn, bz Sponsored by: EMC / Isilon Storage Division
2015-01-05 09:58:32 +00:00
M_ALIGN(n, 0);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
n->m_next = m_split(m, len, wait);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (n->m_next == NULL) {
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
(void) m_free(n);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
return (NULL);
Fix corner case where m_len was not being initialized. Submitted by: Maksim Yevmenkin <myevmenk@digisle.net> MFC after: 1 week
2002-04-12 00:01:50 +00:00
} else {
n->m_len = 0;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
return (n);
Fix corner case where m_len was not being initialized. Submitted by: Maksim Yevmenkin <myevmenk@digisle.net> MFC after: 1 week
2002-04-12 00:01:50 +00:00
}
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
} else
To ease changes to underlying mbuf structure and the mbuf allocator, reduce the knowledge of mbuf layout, and in particular constants such as M_EXT, MLEN, MHLEN, and so on, in mbuf consumers by unifying various alignment utility functions (M_ALIGN(), MH_ALIGN(), MEXT_ALIGN() in a single M_ALIGN() macro, implemented by a now-inlined m_align() function: - Move m_align() from uipc_mbuf.c to mbuf.h; mark as __inline. - Reimplement M_ALIGN(), MH_ALIGN(), and MEXT_ALIGN() using m_align(). - Update consumers around the tree to simply use M_ALIGN(). This change eliminates a number of cases where mbuf consumers must be aware of whether or not mbufs returned by the allocator use external storage, but also assumptions about the size of the returned mbuf. This will make it easier to introduce changes in how we use external storage, as well as features such as variable-size mbufs. Differential Revision: https://reviews.freebsd.org/D1436 Reviewed by: glebius, trasz, gnn, bz Sponsored by: EMC / Isilon Storage Division
2015-01-05 09:58:32 +00:00
M_ALIGN(n, remain);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
} else if (remain == 0) {
n = m->m_next;
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
m->m_next = NULL;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
return (n);
} else {
- Replace compat macros with function calls. - Remove superfluous cleaning of m_len after allocating. Sponsored by: Nginx, Inc.
2013-03-16 08:57:36 +00:00
n = m_get(wait, m->m_type);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (n == NULL)
return (NULL);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
M_ALIGN(n, remain);
}
extpacket:
Step 4.2: start divorce of M_EXT and M_EXTPG They have more differencies than similarities. For now there is lots of code that would check for M_EXT only and work correctly on M_EXTPG buffers, so still carry M_EXT bit together with M_EXTPG. However, prepare some code for explicit check for M_EXTPG. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:37:16 +00:00
if (m->m_flags & (M_EXT|M_EXTPG)) {
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
n->m_data = m->m_data + len;
Mandatory mbuf cluster reference counting and groundwork for UMA based jumbo 9k and jumbo 16k cluster support. All mbuf's with external storage attached are mandatory reference counted. For clusters and jumbo clusters UMA provides the refcnt storage directly. It does not have to be separatly allocated. Any other type of external storage gets its own refcnt allocated from an UMA mbuf refcnt zone instead of normal kernel malloc. The refcount API MEXT_ADD_REF() and MEXT_REM_REF() is no longer publically accessible. The proper m_* functions have to be used. mb_ctor_clust() and mb_dtor_clust() both handle normal 2K as well as 9k and 16k clusters. Clusters and jumbo clusters may be obtained without attaching it immideatly to an mbuf. This is for high performance cluster allocation in network drivers where mbufs are attached after the cluster has been filled. Tested by: rwatson Sponsored by: TCP/IP Optimizations Fundraise 2005
2005-11-02 16:20:36 +00:00
mb_dupcl(n, m);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
} else {
bcopy(mtod(m, caddr_t) + len, mtod(n, caddr_t), remain);
}
n->m_len = remain;
m->m_len = len;
n->m_next = m->m_next;
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
m->m_next = NULL;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
return (n);
}
/*
* Routine to copy from device local memory into mbufs.
Change m_devget()'s outdated and unused `offset' argument to actually mean something: offset into the first mbuf of the target chain before copying the source data over. Make drivers using m_devget() with a first argument "data - ETHER_ALIGN" to use the offset argument to pass ETHER_ALIGN in. The way it was previously done is potentially dangerous if the source data was at the top of a page and the offset caused the previous page to be copied (if the previous page has not yet been appropriately mapped). The old `offset' argument in m_devget() is not used anywhere (it's always 0) and dates back to ~1995 (and earlier?) when support for ethernet trailers existed. With that support gone, it was merely collecting dust. Tested on alpha by: jlemon Partially submitted by: jlemon Reviewed by: jlemon MFC after: 3 weeks
2001-06-20 19:48:35 +00:00
* Note that `off' argument is offset into first mbuf of target chain from
* which to begin copying the data to.
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*/
struct mbuf *
Change m_devget()'s outdated and unused `offset' argument to actually mean something: offset into the first mbuf of the target chain before copying the source data over. Make drivers using m_devget() with a first argument "data - ETHER_ALIGN" to use the offset argument to pass ETHER_ALIGN in. The way it was previously done is potentially dangerous if the source data was at the top of a page and the offset caused the previous page to be copied (if the previous page has not yet been appropriately mapped). The old `offset' argument in m_devget() is not used anywhere (it's always 0) and dates back to ~1995 (and earlier?) when support for ethernet trailers existed. With that support gone, it was merely collecting dust. Tested on alpha by: jlemon Partially submitted by: jlemon Reviewed by: jlemon MFC after: 3 weeks
2001-06-20 19:48:35 +00:00
m_devget(char *buf, int totlen, int off, struct ifnet *ifp,
Generally migrate to ANSI function headers, and remove 'register' use.
2007-05-16 20:41:08 +00:00
void (*copy)(char *from, caddr_t to, u_int len))
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
struct mbuf *m;
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
struct mbuf *top = NULL, **mp = &top;
Change m_devget()'s outdated and unused `offset' argument to actually mean something: offset into the first mbuf of the target chain before copying the source data over. Make drivers using m_devget() with a first argument "data - ETHER_ALIGN" to use the offset argument to pass ETHER_ALIGN in. The way it was previously done is potentially dangerous if the source data was at the top of a page and the offset caused the previous page to be copied (if the previous page has not yet been appropriately mapped). The old `offset' argument in m_devget() is not used anywhere (it's always 0) and dates back to ~1995 (and earlier?) when support for ethernet trailers existed. With that support gone, it was merely collecting dust. Tested on alpha by: jlemon Partially submitted by: jlemon Reviewed by: jlemon MFC after: 3 weeks
2001-06-20 19:48:35 +00:00
int len;
if (off < 0 || off > MHLEN)
return (NULL);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
while (totlen > 0) {
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
if (top == NULL) { /* First one, must be PKTHDR */
if (totlen + off >= MINCLSIZE) {
Mechanically substitute flags from historic mbuf allocator with malloc(9) flags within sys. Exceptions: - sys/contrib not touched - sys/mbuf.h edited manually
2012-12-05 08:04:20 +00:00
m = m_getcl(M_NOWAIT, MT_DATA, M_PKTHDR);
Change m_devget()'s outdated and unused `offset' argument to actually mean something: offset into the first mbuf of the target chain before copying the source data over. Make drivers using m_devget() with a first argument "data - ETHER_ALIGN" to use the offset argument to pass ETHER_ALIGN in. The way it was previously done is potentially dangerous if the source data was at the top of a page and the offset caused the previous page to be copied (if the previous page has not yet been appropriately mapped). The old `offset' argument in m_devget() is not used anywhere (it's always 0) and dates back to ~1995 (and earlier?) when support for ethernet trailers existed. With that support gone, it was merely collecting dust. Tested on alpha by: jlemon Partially submitted by: jlemon Reviewed by: jlemon MFC after: 3 weeks
2001-06-20 19:48:35 +00:00
len = MCLBYTES;
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
} else {
Mechanically substitute flags from historic mbuf allocator with malloc(9) flags within sys. Exceptions: - sys/contrib not touched - sys/mbuf.h edited manually
2012-12-05 08:04:20 +00:00
m = m_gethdr(M_NOWAIT, MT_DATA);
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
len = MHLEN;
/* Place initial small packet/header at end of mbuf */
In m_devget(), if the data fits in a packet header mbuf, check the amount of data is less than or equal to MHLEN instead of MLEN when placing initial small packet header at end of mbuf. Reviewed by: glebius MFC after: 3 days
2016-09-08 01:02:53 +00:00
if (m && totlen + off + max_linkhdr <= MHLEN) {
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
m->m_data += max_linkhdr;
len -= max_linkhdr;
}
}
if (m == NULL)
return NULL;
m->m_pkthdr.rcvif = ifp;
m->m_pkthdr.len = totlen;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
} else {
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
if (totlen + off >= MINCLSIZE) {
Mechanically substitute flags from historic mbuf allocator with malloc(9) flags within sys. Exceptions: - sys/contrib not touched - sys/mbuf.h edited manually
2012-12-05 08:04:20 +00:00
m = m_getcl(M_NOWAIT, MT_DATA, 0);
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
len = MCLBYTES;
} else {
Mechanically substitute flags from historic mbuf allocator with malloc(9) flags within sys. Exceptions: - sys/contrib not touched - sys/mbuf.h edited manually
2012-12-05 08:04:20 +00:00
m = m_get(M_NOWAIT, MT_DATA);
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
len = MLEN;
}
if (m == NULL) {
m_freem(top);
return NULL;
Change m_devget()'s outdated and unused `offset' argument to actually mean something: offset into the first mbuf of the target chain before copying the source data over. Make drivers using m_devget() with a first argument "data - ETHER_ALIGN" to use the offset argument to pass ETHER_ALIGN in. The way it was previously done is potentially dangerous if the source data was at the top of a page and the offset caused the previous page to be copied (if the previous page has not yet been appropriately mapped). The old `offset' argument in m_devget() is not used anywhere (it's always 0) and dates back to ~1995 (and earlier?) when support for ethernet trailers existed. With that support gone, it was merely collecting dust. Tested on alpha by: jlemon Partially submitted by: jlemon Reviewed by: jlemon MFC after: 3 weeks
2001-06-20 19:48:35 +00:00
}
}
if (off) {
m->m_data += off;
len -= off;
off = 0;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
Change m_devget()'s outdated and unused `offset' argument to actually mean something: offset into the first mbuf of the target chain before copying the source data over. Make drivers using m_devget() with a first argument "data - ETHER_ALIGN" to use the offset argument to pass ETHER_ALIGN in. The way it was previously done is potentially dangerous if the source data was at the top of a page and the offset caused the previous page to be copied (if the previous page has not yet been appropriately mapped). The old `offset' argument in m_devget() is not used anywhere (it's always 0) and dates back to ~1995 (and earlier?) when support for ethernet trailers existed. With that support gone, it was merely collecting dust. Tested on alpha by: jlemon Partially submitted by: jlemon Reviewed by: jlemon MFC after: 3 weeks
2001-06-20 19:48:35 +00:00
m->m_len = len = min(totlen, len);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
if (copy)
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
copy(buf, mtod(m, caddr_t), (u_int)len);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
else
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
bcopy(buf, mtod(m, caddr_t), (u_int)len);
Change m_devget()'s outdated and unused `offset' argument to actually mean something: offset into the first mbuf of the target chain before copying the source data over. Make drivers using m_devget() with a first argument "data - ETHER_ALIGN" to use the offset argument to pass ETHER_ALIGN in. The way it was previously done is potentially dangerous if the source data was at the top of a page and the offset caused the previous page to be copied (if the previous page has not yet been appropriately mapped). The old `offset' argument in m_devget() is not used anywhere (it's always 0) and dates back to ~1995 (and earlier?) when support for ethernet trailers existed. With that support gone, it was merely collecting dust. Tested on alpha by: jlemon Partially submitted by: jlemon Reviewed by: jlemon MFC after: 3 weeks
2001-06-20 19:48:35 +00:00
buf += len;
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*mp = m;
mp = &m->m_next;
totlen -= len;
}
return (top);
}
Moved m_copyback into uipc_mbuf.c
1994-10-04 06:50:01 +00:00
/*
* Copy data from a buffer back into the indicated mbuf chain,
* starting "off" bytes from the beginning, extending the mbuf
* chain if necessary.
*/
void
constify the last argument of m_copyback.
2004-04-18 13:01:28 +00:00
m_copyback(struct mbuf *m0, int off, int len, c_caddr_t cp)
Moved m_copyback into uipc_mbuf.c
1994-10-04 06:50:01 +00:00
{
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
int mlen;
struct mbuf *m = m0, *n;
Moved m_copyback into uipc_mbuf.c
1994-10-04 06:50:01 +00:00
int totlen = 0;
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (m0 == NULL)
Moved m_copyback into uipc_mbuf.c
1994-10-04 06:50:01 +00:00
return;
while (off > (mlen = m->m_len)) {
off -= mlen;
totlen += mlen;
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (m->m_next == NULL) {
Mechanically substitute flags from historic mbuf allocator with malloc(9) flags within sys. Exceptions: - sys/contrib not touched - sys/mbuf.h edited manually
2012-12-05 08:04:20 +00:00
n = m_get(M_NOWAIT, m->m_type);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (n == NULL)
Moved m_copyback into uipc_mbuf.c
1994-10-04 06:50:01 +00:00
goto out;
Bring in mbuma to replace mballoc. mbuma is an Mbuf & Cluster allocator built on top of a number of extensions to the UMA framework, all included herein. Extensions to UMA worth noting: - Better layering between slab <-> zone caches; introduce Keg structure which splits off slab cache away from the zone structure and allows multiple zones to be stacked on top of a single Keg (single type of slab cache); perhaps we should look into defining a subset API on top of the Keg for special use by malloc(9), for example. - UMA_ZONE_REFCNT zones can now be added, and reference counters automagically allocated for them within the end of the associated slab structures. uma_find_refcnt() does a kextract to fetch the slab struct reference from the underlying page, and lookup the corresponding refcnt. mbuma things worth noting: - integrates mbuf & cluster allocations with extended UMA and provides caches for commonly-allocated items; defines several zones (two primary, one secondary) and two kegs. - change up certain code paths that always used to do: m_get() + m_clget() to instead just use m_getcl() and try to take advantage of the newly defined secondary Packet zone. - netstat(1) and systat(1) quickly hacked up to do basic stat reporting but additional stats work needs to be done once some other details within UMA have been taken care of and it becomes clearer to how stats will work within the modified framework. From the user perspective, one implication is that the NMBCLUSTERS compile-time option is no longer used. The maximum number of clusters is still capped off according to maxusers, but it can be made unlimited by setting the kern.ipc.nmbclusters boot-time tunable to zero. Work should be done to write an appropriate sysctl handler allowing dynamic tuning of kern.ipc.nmbclusters at runtime. Additional things worth noting/known issues (READ): - One report of 'ips' (ServeRAID) driver acting really slow in conjunction with mbuma. Need more data. Latest report is that ips is equally sucking with and without mbuma. - Giant leak in NFS code sometimes occurs, can't reproduce but currently analyzing; brueffer is able to reproduce but THIS IS NOT an mbuma-specific problem and currently occurs even WITHOUT mbuma. - Issues in network locking: there is at least one code path in the rip code where one or more locks are acquired and we end up in m_prepend() with M_WAITOK, which causes WITNESS to whine from within UMA. Current temporary solution: force all UMA allocations to be M_NOWAIT from within UMA for now to avoid deadlocks unless WITNESS is defined and we can determine with certainty that we're not holding any locks when we're M_WAITOK. - I've seen at least one weird socketbuffer empty-but- mbuf-still-attached panic. I don't believe this to be related to mbuma but please keep your eyes open, turn on debugging, and capture crash dumps. This change removes more code than it adds. A paper is available detailing the change and considering various performance issues, it was presented at BSDCan2004: http://www.unixdaemons.com/~bmilekic/netbuf_bmilekic.pdf Please read the paper for Future Work and implementation details, as well as credits. Testing and Debugging: rwatson, brueffer, Ketrien I. Saihr-Kesenchedra, ... Reviewed by: Lots of people (for different parts)
2004-05-31 21:46:06 +00:00
bzero(mtod(n, caddr_t), MLEN);
Moved m_copyback into uipc_mbuf.c
1994-10-04 06:50:01 +00:00
n->m_len = min(MLEN, len + off);
m->m_next = n;
}
m = m->m_next;
}
while (len > 0) {
Teach m_copyback() to use trailing space of the last mbuf in chain.
2009-01-18 20:19:55 +00:00
if (m->m_next == NULL && (len > m->m_len - off)) {
m->m_len += min(len - (m->m_len - off),
M_TRAILINGSPACE(m));
}
Moved m_copyback into uipc_mbuf.c
1994-10-04 06:50:01 +00:00
mlen = min (m->m_len - off, len);
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
bcopy(cp, off + mtod(m, caddr_t), (u_int)mlen);
Moved m_copyback into uipc_mbuf.c
1994-10-04 06:50:01 +00:00
cp += mlen;
len -= mlen;
mlen += off;
off = 0;
totlen += mlen;
if (len == 0)
break;
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (m->m_next == NULL) {
Mechanically substitute flags from historic mbuf allocator with malloc(9) flags within sys. Exceptions: - sys/contrib not touched - sys/mbuf.h edited manually
2012-12-05 08:04:20 +00:00
n = m_get(M_NOWAIT, m->m_type);
Long awaited style fixup in mbuf code. Get rid of K&R style prototyping and function argument declarations. Make sure that functions that are supposed to return a pointer return NULL in case of failure. Don't cast NULL. Finally, get rid of annoying `register' uses.
2001-02-11 05:02:06 +00:00
if (n == NULL)
Moved m_copyback into uipc_mbuf.c
1994-10-04 06:50:01 +00:00
break;
n->m_len = min(MLEN, len);
m->m_next = n;
}
m = m->m_next;
}
out: if (((m = m0)->m_flags & M_PKTHDR) && (m->m_pkthdr.len < totlen))
m->m_pkthdr.len = totlen;
}
New function: m_print(struct mbuf *); hexdumps a mbuf.
1999-11-01 15:03:20 +00:00
add m_append utility function to be used in forthcoming changes
2004-12-08 05:42:02 +00:00
/*
* Append the specified data to the indicated mbuf chain,
* Extend the mbuf chain if the new data does not fit in
* existing space.
*
* Return 1 if able to complete the job; otherwise 0.
*/
int
m_append(struct mbuf *m0, int len, c_caddr_t cp)
{
struct mbuf *m, *n;
int remainder, space;
for (m = m0; m->m_next != NULL; m = m->m_next)
;
remainder = len;
space = M_TRAILINGSPACE(m);
if (space > 0) {
/*
* Copy into available space.
*/
if (space > remainder)
space = remainder;
bcopy(cp, mtod(m, caddr_t) + m->m_len, space);
m->m_len += space;
cp += space, remainder -= space;
}
while (remainder > 0) {
/*
* Allocate a new mbuf; could check space
* and allocate a cluster instead.
*/
Mechanically substitute flags from historic mbuf allocator with malloc(9) flags within sys. Exceptions: - sys/contrib not touched - sys/mbuf.h edited manually
2012-12-05 08:04:20 +00:00
n = m_get(M_NOWAIT, m->m_type);
add m_append utility function to be used in forthcoming changes
2004-12-08 05:42:02 +00:00
if (n == NULL)
break;
n->m_len = min(MLEN, remainder);
fix m_append for case where additional mbufs are required
2004-12-15 19:04:07 +00:00
bcopy(cp, mtod(n, caddr_t), n->m_len);
cp += n->m_len, remainder -= n->m_len;
add m_append utility function to be used in forthcoming changes
2004-12-08 05:42:02 +00:00
m->m_next = n;
m = n;
}
if (m0->m_flags & M_PKTHDR)
m0->m_pkthdr.len += len - remainder;
return (remainder == 0);
}
Push m_apply() and m_getptr() up into the colleciton of standard mbuf routines, and purge them from opencrypto. Reviewed by: sam Obtained from: NetBSD Sponsored by: spc.org
2003-12-15 21:49:41 +00:00
/*
* Apply function f to the data in an mbuf chain starting "off" bytes from
* the beginning, continuing for "len" bytes.
*/
int
m_apply(struct mbuf *m, int off, int len,
style(9) pass and type fixups. Submitted by: bde
2003-12-16 14:13:47 +00:00
int (*f)(void *, void *, u_int), void *arg)
Push m_apply() and m_getptr() up into the colleciton of standard mbuf routines, and purge them from opencrypto. Reviewed by: sam Obtained from: NetBSD Sponsored by: spc.org
2003-12-15 21:49:41 +00:00
{
style(9) pass and type fixups. Submitted by: bde
2003-12-16 14:13:47 +00:00
u_int count;
Push m_apply() and m_getptr() up into the colleciton of standard mbuf routines, and purge them from opencrypto. Reviewed by: sam Obtained from: NetBSD Sponsored by: spc.org
2003-12-15 21:49:41 +00:00
int rval;
KASSERT(off >= 0, ("m_apply, negative off %d", off));
KASSERT(len >= 0, ("m_apply, negative len %d", len));
while (off > 0) {
KASSERT(m != NULL, ("m_apply, offset > size of mbuf chain"));
if (off < m->m_len)
break;
off -= m->m_len;
m = m->m_next;
}
while (len > 0) {
KASSERT(m != NULL, ("m_apply, offset > size of mbuf chain"));
count = min(m->m_len - off, len);
rval = (*f)(arg, mtod(m, caddr_t) + off, count);
if (rval)
return (rval);
len -= count;
off = 0;
m = m->m_next;
}
return (0);
}
/*
* Return a pointer to mbuf/offset of location in mbuf chain.
*/
struct mbuf *
m_getptr(struct mbuf *m, int loc, int *off)
{
while (loc >= 0) {
style(9) pass and type fixups. Submitted by: bde
2003-12-16 14:13:47 +00:00
/* Normal end of search. */
Push m_apply() and m_getptr() up into the colleciton of standard mbuf routines, and purge them from opencrypto. Reviewed by: sam Obtained from: NetBSD Sponsored by: spc.org
2003-12-15 21:49:41 +00:00
if (m->m_len > loc) {
*off = loc;
return (m);
} else {
loc -= m->m_len;
if (m->m_next == NULL) {
if (loc == 0) {
style(9) pass and type fixups. Submitted by: bde
2003-12-16 14:13:47 +00:00
/* Point at the end of valid data. */
Push m_apply() and m_getptr() up into the colleciton of standard mbuf routines, and purge them from opencrypto. Reviewed by: sam Obtained from: NetBSD Sponsored by: spc.org
2003-12-15 21:49:41 +00:00
*off = m->m_len;
return (m);
style(9) pass and type fixups. Submitted by: bde
2003-12-16 14:13:47 +00:00
}
return (NULL);
}
m = m->m_next;
Push m_apply() and m_getptr() up into the colleciton of standard mbuf routines, and purge them from opencrypto. Reviewed by: sam Obtained from: NetBSD Sponsored by: spc.org
2003-12-15 21:49:41 +00:00
}
}
return (NULL);
}
New function: m_print(struct mbuf *); hexdumps a mbuf.
1999-11-01 15:03:20 +00:00
void
improve the mbuf m_print function.. Only pull length from pkthdr if there is one, detect mbuf loops and stop, add an extra arg so you can only print the first x bytes of the data per mbuf (print all if arg is -1), print flags using %b (bitmask)... No code in the tree appears to use m_print, and it's just a maner of adding -1 as an additional arg to m_print to restore original behavior.. MFC after: 4 days
2004-09-28 18:40:18 +00:00
m_print(const struct mbuf *m, int maxlen)
New function: m_print(struct mbuf *); hexdumps a mbuf.
1999-11-01 15:03:20 +00:00
{
int len;
improve the mbuf m_print function.. Only pull length from pkthdr if there is one, detect mbuf loops and stop, add an extra arg so you can only print the first x bytes of the data per mbuf (print all if arg is -1), print flags using %b (bitmask)... No code in the tree appears to use m_print, and it's just a maner of adding -1 as an additional arg to m_print to restore original behavior.. MFC after: 4 days
2004-09-28 18:40:18 +00:00
int pdata;
Make m_print const correct (avoids a warning)
1999-12-20 18:10:00 +00:00
const struct mbuf *m2;
New function: m_print(struct mbuf *); hexdumps a mbuf.
1999-11-01 15:03:20 +00:00
Xen netback driver rewrite. share/man/man4/Makefile, share/man/man4/xnb.4, sys/dev/xen/netback/netback.c, sys/dev/xen/netback/netback_unit_tests.c: Rewrote the netback driver for xen to attach properly via newbus and work properly in both HVM and PVM mode (only HVM is tested). Works with the in-tree FreeBSD netfront driver or the Windows netfront driver from SuSE. Has not been extensively tested with a Linux netfront driver. Does not implement LRO, TSO, or polling. Includes unit tests that may be run through sysctl after compiling with XNB_DEBUG defined. sys/dev/xen/blkback/blkback.c, sys/xen/interface/io/netif.h: Comment elaboration. sys/kern/uipc_mbuf.c: Fix page fault in kernel mode when calling m_print() on a null mbuf. Since m_print() is only used for debugging, there are no performance concerns for extra error checking code. sys/kern/subr_scanf.c: Add the "hh" and "ll" width specifiers from C99 to scanf(). A few callers were already using "ll" even though scanf() was handling it as "l". Submitted by: Alan Somers <alans@spectralogic.com> Submitted by: John Suykerbuyk <johns@spectralogic.com> Sponsored by: Spectra Logic MFC after: 1 week Reviewed by: ken
2012-01-26 16:35:09 +00:00
if (m == NULL) {
printf("mbuf: %p\n", m);
return;
}
improve the mbuf m_print function.. Only pull length from pkthdr if there is one, detect mbuf loops and stop, add an extra arg so you can only print the first x bytes of the data per mbuf (print all if arg is -1), print flags using %b (bitmask)... No code in the tree appears to use m_print, and it's just a maner of adding -1 as an additional arg to m_print to restore original behavior.. MFC after: 4 days
2004-09-28 18:40:18 +00:00
if (m->m_flags & M_PKTHDR)
len = m->m_pkthdr.len;
else
len = -1;
New function: m_print(struct mbuf *); hexdumps a mbuf.
1999-11-01 15:03:20 +00:00
m2 = m;
improve the mbuf m_print function.. Only pull length from pkthdr if there is one, detect mbuf loops and stop, add an extra arg so you can only print the first x bytes of the data per mbuf (print all if arg is -1), print flags using %b (bitmask)... No code in the tree appears to use m_print, and it's just a maner of adding -1 as an additional arg to m_print to restore original behavior.. MFC after: 4 days
2004-09-28 18:40:18 +00:00
while (m2 != NULL && (len == -1 || len)) {
pdata = m2->m_len;
if (maxlen != -1 && pdata > maxlen)
pdata = maxlen;
printf("mbuf: %p len: %d, next: %p, %b%s", m2, m2->m_len,
m2->m_next, m2->m_flags, "\20\20freelist\17skipfw"
"\11proto5\10proto4\7proto3\6proto2\5proto1\4rdonly"
"\3eor\2pkthdr\1ext", pdata ? "" : "\n");
if (pdata)
spell pdata correctly, we now will only dump maxlen of each mbuf in the chain, instead of the entire mbuf... This should probably be reworked so that it prints at max maxlen bytes for the entire chain...
2006-03-14 00:22:10 +00:00
printf(", %*D\n", pdata, (u_char *)m2->m_data, "-");
improve the mbuf m_print function.. Only pull length from pkthdr if there is one, detect mbuf loops and stop, add an extra arg so you can only print the first x bytes of the data per mbuf (print all if arg is -1), print flags using %b (bitmask)... No code in the tree appears to use m_print, and it's just a maner of adding -1 as an additional arg to m_print to restore original behavior.. MFC after: 4 days
2004-09-28 18:40:18 +00:00
if (len != -1)
len -= m2->m_len;
New function: m_print(struct mbuf *); hexdumps a mbuf.
1999-11-01 15:03:20 +00:00
m2 = m2->m_next;
}
improve the mbuf m_print function.. Only pull length from pkthdr if there is one, detect mbuf loops and stop, add an extra arg so you can only print the first x bytes of the data per mbuf (print all if arg is -1), print flags using %b (bitmask)... No code in the tree appears to use m_print, and it's just a maner of adding -1 as an additional arg to m_print to restore original behavior.. MFC after: 4 days
2004-09-28 18:40:18 +00:00
if (len > 0)
printf("%d bytes unaccounted for.\n", len);
New function: m_print(struct mbuf *); hexdumps a mbuf.
1999-11-01 15:03:20 +00:00
return;
}
Move m_fixhdr() from "mbchain" to "mbuf" where it belongs.
2002-09-18 13:41:37 +00:00
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
u_int
Move m_fixhdr() from "mbchain" to "mbuf" where it belongs.
2002-09-18 13:41:37 +00:00
m_fixhdr(struct mbuf *m0)
{
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
u_int len;
Move m_fixhdr() from "mbchain" to "mbuf" where it belongs.
2002-09-18 13:41:37 +00:00
Introduce the m_length() function which will return the accumulated length of an mbuf-chain and optionally a pointer to the last mbuf.
2002-09-18 14:57:35 +00:00
len = m_length(m0, NULL);
m0->m_pkthdr.len = len;
return (len);
}
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
u_int
Introduce the m_length() function which will return the accumulated length of an mbuf-chain and optionally a pointer to the last mbuf.
2002-09-18 14:57:35 +00:00
m_length(struct mbuf *m0, struct mbuf **last)
{
struct mbuf *m;
style nit: unsigned -> u_int in the kernel, particularly to stay consistent in this file, and keep m_length() and m_fixhdr() consistent with their prototypes in mbuf.h Inspired by: bde
2002-09-18 22:33:52 +00:00
u_int len;
Introduce the m_length() function which will return the accumulated length of an mbuf-chain and optionally a pointer to the last mbuf.
2002-09-18 14:57:35 +00:00
len = 0;
for (m = m0; m != NULL; m = m->m_next) {
Move m_fixhdr() from "mbchain" to "mbuf" where it belongs.
2002-09-18 13:41:37 +00:00
len += m->m_len;
Introduce the m_length() function which will return the accumulated length of an mbuf-chain and optionally a pointer to the last mbuf.
2002-09-18 14:57:35 +00:00
if (m->m_next == NULL)
break;
Move m_fixhdr() from "mbchain" to "mbuf" where it belongs.
2002-09-18 13:41:37 +00:00
}
Introduce the m_length() function which will return the accumulated length of an mbuf-chain and optionally a pointer to the last mbuf.
2002-09-18 14:57:35 +00:00
if (last != NULL)
*last = m;
return (len);
Move m_fixhdr() from "mbchain" to "mbuf" where it belongs.
2002-09-18 13:41:37 +00:00
}
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
/*
* Defragment a mbuf chain, returning the shortest possible
* chain of mbufs and clusters. If allocation fails and
* this cannot be completed, NULL will be returned, but
* the passed in chain will be unchanged. Upon success,
* the original chain will be freed, and the new chain
* will be returned.
*
* If a non-packet header is passed in, the original
* mbuf (chain?) will be returned unharmed.
*/
struct mbuf *
m_defrag(struct mbuf *m0, int how)
{
Style fixes: don't indent variable names. Submitted by: bde
2004-02-05 08:29:27 +00:00
struct mbuf *m_new = NULL, *m_final = NULL;
int progress = 0, length;
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
Make sure we don't call mbuf allocation functions with mutexes held. Discussed with: rwatson
2004-07-21 07:12:24 +00:00
MBUF_CHECKSLEEP(how);
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
if (!(m0->m_flags & M_PKTHDR))
return (m0);
Three fixes: - Make m_prepend use m_gethdr instead of m_get where appropriate - Make m_copym use m_gethdr instead of m_get where appropriate - Add a call to m_fixhdr in m_defrag; m_defrag can't deal with corrupted pkthdr.len counts. MFC after: 3 days
2003-07-19 06:03:48 +00:00
m_fixhdr(m0); /* Needed sanity check */
Add another MBUF_STRESS_TEST feature, m_defragrandomfailures. When enabled, this causes m_defrag to randomly return NULL (following its normal failure case so that extra memory leaks are not introduced.) Code similar to this was used to find / fix a few bugs last week.
2003-04-15 02:14:43 +00:00
#ifdef MBUF_STRESS_TEST
if (m_defragrandomfailures) {
int temp = arc4random() & 0xff;
if (temp == 0xba)
goto nospace;
}
#endif
Whitespace only.
2014-05-30 08:22:58 +00:00
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
if (m0->m_pkthdr.len > MHLEN)
m_final = m_getcl(how, MT_DATA, M_PKTHDR);
else
m_final = m_gethdr(how, MT_DATA);
if (m_final == NULL)
goto nospace;
Catch a few places where NULL (pointer) was used where 0 (integer) was expected.
2003-12-23 02:36:43 +00:00
if (m_dup_pkthdr(m_final, m0, how) == 0)
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
goto nospace;
m_new = m_final;
while (progress < m0->m_pkthdr.len) {
length = m0->m_pkthdr.len - progress;
if (length > MCLBYTES)
length = MCLBYTES;
if (m_new == NULL) {
if (length > MLEN)
m_new = m_getcl(how, MT_DATA, 0);
else
m_new = m_get(how, MT_DATA);
if (m_new == NULL)
goto nospace;
}
m_copydata(m0, progress, length, mtod(m_new, caddr_t));
progress += length;
m_new->m_len = length;
if (m_new != m_final)
m_cat(m_final, m_new);
m_new = NULL;
}
Hide the m_defrag* statistics under MBUF_STRESS_TEST, there seems to be no need to see them in the general case (and they aren't smp-safe anyway.) Suggested by: hmp MFC after: 1 week
2003-06-17 02:34:40 +00:00
#ifdef MBUF_STRESS_TEST
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
if (m0->m_next == NULL)
m_defraguseless++;
Hide the m_defrag* statistics under MBUF_STRESS_TEST, there seems to be no need to see them in the general case (and they aren't smp-safe anyway.) Suggested by: hmp MFC after: 1 week
2003-06-17 02:34:40 +00:00
#endif
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
m_freem(m0);
m0 = m_final;
Hide the m_defrag* statistics under MBUF_STRESS_TEST, there seems to be no need to see them in the general case (and they aren't smp-safe anyway.) Suggested by: hmp MFC after: 1 week
2003-06-17 02:34:40 +00:00
#ifdef MBUF_STRESS_TEST
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
m_defragpackets++;
m_defragbytes += m0->m_pkthdr.len;
Hide the m_defrag* statistics under MBUF_STRESS_TEST, there seems to be no need to see them in the general case (and they aren't smp-safe anyway.) Suggested by: hmp MFC after: 1 week
2003-06-17 02:34:40 +00:00
#endif
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
return (m0);
nospace:
Hide the m_defrag* statistics under MBUF_STRESS_TEST, there seems to be no need to see them in the general case (and they aren't smp-safe anyway.) Suggested by: hmp MFC after: 1 week
2003-06-17 02:34:40 +00:00
#ifdef MBUF_STRESS_TEST
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
m_defragfailure++;
Hide the m_defrag* statistics under MBUF_STRESS_TEST, there seems to be no need to see them in the general case (and they aren't smp-safe anyway.) Suggested by: hmp MFC after: 1 week
2003-06-17 02:34:40 +00:00
#endif
Add the m_defrag routine, as discussed on committers@. This incarnation should address the concerns of all in the discussion, and keeps statistics which show how much it is used. MFC after: 2 weeks
2003-03-29 05:48:36 +00:00
if (m_final)
m_freem(m_final);
return (NULL);
}
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
/*
* Return the number of fragments an mbuf will use. This is usually
* used as a proxy for the number of scatter/gather elements needed by
* a DMA engine to access an mbuf. In general mapped mbufs are
* assumed to be backed by physically contiguous buffers that only
* need a single fragment. Unmapped mbufs, on the other hand, can
* span disjoint physical pages.
*/
static int
frags_per_mbuf(struct mbuf *m)
{
int frags;
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
if ((m->m_flags & M_EXTPG) == 0)
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
return (1);
/*
* The header and trailer are counted as a single fragment
* each when present.
*
* XXX: This overestimates the number of fragments by assuming
* all the backing physical pages are disjoint.
*/
frags = 0;
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
if (m->m_epg_hdrlen != 0)
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
frags++;
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
frags += m->m_epg_npgs;
if (m->m_epg_trllen != 0)
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
frags++;
return (frags);
}
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
2008-01-17 21:25:09 +00:00
/*
* Defragment an mbuf chain, returning at most maxfrags separate
* mbufs+clusters. If this is not possible NULL is returned and
Fix improper use of "its". Sponsored by: Dell EMC Isilon
2016-11-08 23:59:41 +00:00
* the original mbuf chain is left in its present (potentially
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
2008-01-17 21:25:09 +00:00
* modified) state. We use two techniques: collapsing consecutive
* mbufs and replacing consecutive mbufs by a cluster.
*
* NB: this should really be named m_defrag but that name is taken
*/
struct mbuf *
m_collapse(struct mbuf *m0, int how, int maxfrags)
{
struct mbuf *m, *n, *n2, **prev;
u_int curfrags;
/*
* Calculate the current number of frags.
*/
curfrags = 0;
for (m = m0; m != NULL; m = m->m_next)
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
curfrags += frags_per_mbuf(m);
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
2008-01-17 21:25:09 +00:00
/*
* First, try to collapse mbufs. Note that we always collapse
* towards the front so we don't need to deal with moving the
* pkthdr. This may be suboptimal if the first mbuf has much
* less data than the following.
*/
m = m0;
again:
for (;;) {
n = m->m_next;
if (n == NULL)
break;
Improve m_cat() by being able to also merge contents from M_EXT mbuf's by doing proper testing with M_WRITABLE(). In m_collapse() replace an incomplete manual check for M_RDONLY with the M_WRITABLE() macro that also tests for shared buffers and other cases that make a particular mbuf immutable. MFC after: 2 weeks
2012-10-28 18:38:51 +00:00
if (M_WRITABLE(m) &&
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
2008-01-17 21:25:09 +00:00
n->m_len < M_TRAILINGSPACE(m)) {
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
m_copydata(n, 0, n->m_len,
mtod(m, char *) + m->m_len);
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
2008-01-17 21:25:09 +00:00
m->m_len += n->m_len;
m->m_next = n->m_next;
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
curfrags -= frags_per_mbuf(n);
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
2008-01-17 21:25:09 +00:00
m_free(n);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
if (curfrags <= maxfrags)
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
2008-01-17 21:25:09 +00:00
return m0;
} else
m = n;
}
KASSERT(maxfrags > 1,
("maxfrags %u, but normal collapse failed", maxfrags));
/*
* Collapse consecutive mbufs to a cluster.
*/
prev = &m0->m_next; /* NB: not the first mbuf */
while ((n = *prev) != NULL) {
if ((n2 = n->m_next) != NULL &&
n->m_len + n2->m_len < MCLBYTES) {
m = m_getcl(how, MT_DATA, 0);
if (m == NULL)
goto bad;
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
m_copydata(n, 0, n->m_len, mtod(m, char *));
m_copydata(n2, 0, n2->m_len,
mtod(m, char *) + n->m_len);
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
2008-01-17 21:25:09 +00:00
m->m_len = n->m_len + n2->m_len;
m->m_next = n2->m_next;
*prev = m;
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
curfrags += 1; /* For the new cluster */
curfrags -= frags_per_mbuf(n);
curfrags -= frags_per_mbuf(n2);
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
2008-01-17 21:25:09 +00:00
m_free(n);
m_free(n2);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
if (curfrags <= maxfrags)
promote ath_defrag to m_collapse (and retire private+unused m_collapse from cxgb) Reviewed by: pyun, jhb, kmacy MFC after: 2 weeks
2008-01-17 21:25:09 +00:00
return m0;
/*
* Still not there, try the normal collapse
* again before we allocate another cluster.
*/
goto again;
}
prev = &n->m_next;
}
/*
* No place where we can collapse to a cluster; punt.
* This can occur if, for example, you request 2 frags
* but the packet requires that both be clusters (we
* never reallocate the first mbuf to avoid moving the
* packet header).
*/
bad:
return NULL;
}
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
#ifdef MBUF_STRESS_TEST
/*
* Fragment an mbuf chain. There's no reason you'd ever want to do
* this in normal usage, but it's great for stress testing various
* mbuf consumers.
*
* If fragmentation is not possible, the original chain will be
* returned.
*
* Possible length values:
* 0 no fragmentation will occur
* > 0 each fragment will be of the specified length
* -1 each fragment will be the same random value in length
* -2 each fragment's length will be entirely random
* (Random values range from 1 to 256)
*/
struct mbuf *
m_fragment(struct mbuf *m0, int how, int length)
{
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
struct mbuf *m_first, *m_last;
int divisor = 255, progress = 0, fraglen;
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
if (!(m0->m_flags & M_PKTHDR))
return (m0);
Whitespace only.
2014-05-30 08:22:58 +00:00
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
if (length == 0 || length < -2)
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
return (m0);
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
if (length > MCLBYTES)
length = MCLBYTES;
if (length < 0 && divisor > MCLBYTES)
divisor = MCLBYTES;
if (length == -1)
length = 1 + (arc4random() % divisor);
if (length > 0)
fraglen = length;
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
m_fixhdr(m0); /* Needed sanity check */
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
m_first = m_getcl(how, MT_DATA, M_PKTHDR);
if (m_first == NULL)
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
goto nospace;
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
if (m_dup_pkthdr(m_first, m0, how) == 0)
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
goto nospace;
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
m_last = m_first;
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
while (progress < m0->m_pkthdr.len) {
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
if (length == -2)
fraglen = 1 + (arc4random() % divisor);
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
if (fraglen > m0->m_pkthdr.len - progress)
fraglen = m0->m_pkthdr.len - progress;
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
if (progress != 0) {
struct mbuf *m_new = m_getcl(how, MT_DATA, 0);
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
if (m_new == NULL)
goto nospace;
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
m_last->m_next = m_new;
m_last = m_new;
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
}
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
m_copydata(m0, progress, fraglen, mtod(m_last, caddr_t));
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
progress += fraglen;
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
m_last->m_len = fraglen;
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
}
m_freem(m0);
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
m0 = m_first;
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
return (m0);
nospace:
mbuf(9): unbreak m_fragment() - Fix it by replacing m_cat() with m_prev->m_next = m_new (m_cat() will try to append data - as a result, there will be no fragmentation). - Move some constants out of the loop. Was previously tested with D4077. Differential Revision: https://reviews.freebsd.org/D4090
2017-10-16 21:46:11 +00:00
if (m_first)
m_freem(m_first);
Implement MBUF_STRESS_TEST mark II. Changes from the original implementation: - Fragmentation is handled by the function m_fragment, which can be called from whereever fragmentation is needed. Note that this function is wrapped in #ifdef MBUF_STRESS_TEST to discourage non-testing use. - m_fragment works slightly differently from the old fragmentation code in that it allocates a seperate mbuf cluster for each fragment. This defeats dma_map_load_mbuf/buffer's feature of coalescing adjacent fragments. While that is a nice feature in practice, it nerfed the usefulness of mbuf_stress_test. - Add two modes of random fragmentation. Chains with fragments all of the same random length and chains with fragments that are each uniquely random in length may now be requested.
2003-09-01 05:55:37 +00:00
/* Return the original chain on failure */
return (m0);
}
#endif
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
/*
* Free pages from mbuf_ext_pgs, assuming they were allocated via
* vm_page_alloc() and aren't associated with any object. Complement
* to allocator from m_uiotombuf_nomap().
*/
void
mb_free_mext_pgs(struct mbuf *m)
{
vm_page_t pg;
Mechanically rename MBUF_EXT_PGS_ASSERT() to M_ASSERTEXTPG() to match classical M_ASSERTPKTHDR. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:27:41 +00:00
M_ASSERTEXTPG(m);
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
for (int i = 0; i < m->m_epg_npgs; i++) {
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
pg = PHYS_TO_VM_PAGE(m->m_epg_pa[i]);
Remove manual wire_count adjustments from the unmapped mbuf code. The original code came from a desire to minimize the number of updates to v_wire_count, which prior to r329187 was updated using atomics. However, there is no significant benefit to batching today, so simply allocate pages using VM_ALLOC_WIRED and rely on system accounting. Reviewed by: jhb Differential Revision: https://reviews.freebsd.org/D21323
2019-08-21 20:01:52 +00:00
vm_page_unwire_noq(pg);
vm_page_free(pg);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
}
}
static struct mbuf *
m_uiotombuf_nomap(struct uio *uio, int how, int len, int maxseg, int flags)
{
struct mbuf *m, *mb, *prev;
vm_page_t pg_array[MBUF_PEXT_MAX_PGS];
Remove manual wire_count adjustments from the unmapped mbuf code. The original code came from a desire to minimize the number of updates to v_wire_count, which prior to r329187 was updated using atomics. However, there is no significant benefit to batching today, so simply allocate pages using VM_ALLOC_WIRED and rely on system accounting. Reviewed by: jhb Differential Revision: https://reviews.freebsd.org/D21323
2019-08-21 20:01:52 +00:00
int error, length, i, needed;
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
ssize_t total;
Remove manual wire_count adjustments from the unmapped mbuf code. The original code came from a desire to minimize the number of updates to v_wire_count, which prior to r329187 was updated using atomics. However, there is no significant benefit to batching today, so simply allocate pages using VM_ALLOC_WIRED and rely on system accounting. Reviewed by: jhb Differential Revision: https://reviews.freebsd.org/D21323
2019-08-21 20:01:52 +00:00
int pflags = malloc2vm_flags(how) | VM_ALLOC_NOOBJ | VM_ALLOC_NODUMP |
VM_ALLOC_WIRED;
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
/*
* len can be zero or an arbitrary large value bound by
* the total data supplied by the uio.
*/
if (len > 0)
total = MIN(uio->uio_resid, len);
else
total = uio->uio_resid;
if (maxseg == 0)
maxseg = MBUF_PEXT_MAX_PGS * PAGE_SIZE;
/*
* Allocate the pages
*/
m = NULL;
KTLS: Re-work unmapped mbufs to carry ext_pgs in the mbuf itself. While the original implementation of unmapped mbufs was a large step forward in terms of reducing cache misses by enabling mbufs to carry more than a single page for sendfile, they are rather cache unfriendly when accessing the ext_pgs metadata and data. This is because the ext_pgs part of the mbuf is allocated separately, and almost guaranteed to be cold in cache. This change takes advantage of the fact that unmapped mbufs are never used at the same time as pkthdr mbufs. Given this fact, we can overlap the ext_pgs metadata with the mbuf pkthdr, and carry the ext_pgs meta directly in the mbuf itself. Similarly, we can carry the ext_pgs data (TLS hdr/trailer/array of pages) directly after the existing m_ext. In order to be able to carry 5 pages (which is the minimum required for a 16K TLS record which is not perfectly aligned) on LP64, I've had to steal ext_arg2. The only user of this in the xmit path is sendfile, and I've adjusted it to use arg1 when using unmapped mbufs. This change is almost entirely mechanical, except that we change mb_alloc_ext_pgs() to no longer allow allocating pkthdrs, the change to avoid ext_arg2 as mentioned above, and the removal of the ext_pgs zone, This change saves roughly 2% "raw" CPU (~59% -> 57%), or over 3% "scaled" CPU on a Netflix 100% software kTLS workload at 90+ Gb/s on Broadwell Xeons. In a follow-on commit, I plan to remove some hacks to avoid access ext_pgs fields of mbufs, since they will now be in cache. Many thanks to glebius for helping to make this better in the Netflix tree. Reviewed by: hselasky, jhb, rrs, glebius (early version) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D24213
2020-04-14 14:46:06 +00:00
MPASS((flags & M_PKTHDR) == 0);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
while (total > 0) {
KTLS: Re-work unmapped mbufs to carry ext_pgs in the mbuf itself. While the original implementation of unmapped mbufs was a large step forward in terms of reducing cache misses by enabling mbufs to carry more than a single page for sendfile, they are rather cache unfriendly when accessing the ext_pgs metadata and data. This is because the ext_pgs part of the mbuf is allocated separately, and almost guaranteed to be cold in cache. This change takes advantage of the fact that unmapped mbufs are never used at the same time as pkthdr mbufs. Given this fact, we can overlap the ext_pgs metadata with the mbuf pkthdr, and carry the ext_pgs meta directly in the mbuf itself. Similarly, we can carry the ext_pgs data (TLS hdr/trailer/array of pages) directly after the existing m_ext. In order to be able to carry 5 pages (which is the minimum required for a 16K TLS record which is not perfectly aligned) on LP64, I've had to steal ext_arg2. The only user of this in the xmit path is sendfile, and I've adjusted it to use arg1 when using unmapped mbufs. This change is almost entirely mechanical, except that we change mb_alloc_ext_pgs() to no longer allow allocating pkthdrs, the change to avoid ext_arg2 as mentioned above, and the removal of the ext_pgs zone, This change saves roughly 2% "raw" CPU (~59% -> 57%), or over 3% "scaled" CPU on a Netflix 100% software kTLS workload at 90+ Gb/s on Broadwell Xeons. In a follow-on commit, I plan to remove some hacks to avoid access ext_pgs fields of mbufs, since they will now be in cache. Many thanks to glebius for helping to make this better in the Netflix tree. Reviewed by: hselasky, jhb, rrs, glebius (early version) Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D24213
2020-04-14 14:46:06 +00:00
mb = mb_alloc_ext_pgs(how, mb_free_mext_pgs);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
if (mb == NULL)
goto failed;
if (m == NULL)
m = mb;
else
prev->m_next = mb;
prev = mb;
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
mb->m_epg_flags = EPG_FLAG_ANON;
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
needed = length = MIN(maxseg, total);
for (i = 0; needed > 0; i++, needed -= PAGE_SIZE) {
retry_page:
pg_array[i] = vm_page_alloc(NULL, 0, pflags);
if (pg_array[i] == NULL) {
if (how & M_NOWAIT) {
goto failed;
} else {
vm_wait(NULL);
goto retry_page;
}
}
pg_array[i]->flags &= ~PG_ZERO;
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
mb->m_epg_pa[i] = VM_PAGE_TO_PHYS(pg_array[i]);
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
mb->m_epg_npgs++;
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
}
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
mb->m_epg_last_len = length - PAGE_SIZE * (mb->m_epg_npgs - 1);
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
MBUF_EXT_PGS_ASSERT_SANITY(mb);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
total -= length;
error = uiomove_fromphys(pg_array, 0, length, uio);
if (error != 0)
goto failed;
mb->m_len = length;
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
mb->m_ext.ext_size += PAGE_SIZE * mb->m_epg_npgs;
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
if (flags & M_PKTHDR)
m->m_pkthdr.len += length;
}
return (m);
failed:
m_freem(m);
return (NULL);
}
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
/*
* Copy the contents of uio into a properly sized mbuf chain.
*/
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
struct mbuf *
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
m_uiotombuf(struct uio *uio, int how, int len, int align, int flags)
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
{
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
struct mbuf *m, *mb;
Fix found places where uio_resid is truncated to int. Add the sysctl debug.iosize_max_clamp, enabled by default. Setting the sysctl to zero allows to perform the SSIZE_MAX-sized i/o requests from the usermode. Discussed with: bde, das (previous versions) MFC after: 1 month
2012-02-21 01:05:12 +00:00
int error, length;
ssize_t total;
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
int progress = 0;
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
if (flags & M_EXTPG)
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
return (m_uiotombuf_nomap(uio, how, len, align, flags));
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
/*
* len can be zero or an arbitrary large value bound by
* the total data supplied by the uio.
*/
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
if (len > 0)
Fix one more place uio_resid is truncated to int A follow-up to r231949 and r194990. Reported by: pho@ Reviewed by: kib@, markj@ Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D11373
2017-06-27 17:23:20 +00:00
total = (uio->uio_resid < len) ? uio->uio_resid : len;
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
else
total = uio->uio_resid;
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
/*
* The smallest unit returned by m_getm2() is a single mbuf
Remove the alignment of the align parameter. This is up to the caller to pass in and it breaks tap(4) on strict alignment machines as m_uiotombuf is called with ETHER_ALIGN. Found by: Jared Go Reviewed by: emax MFC after: 3 days
2008-09-05 04:05:31 +00:00
* with pkthdr. We can't align past it.
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
*/
Change m_uiotombuf so it will accept offset at which data should be copied to the mbuf. Offset cannot exceed MHLEN bytes. This is currently used to fix Ethernet header alignment problem on alpha and sparc64. Also change all users of m_uiotombuf to pass proper offset. Reviewed by: jmg, sam Tested by: Sten Spans "sten AT blinkenlights DOT nl" MFC after: 1 week
2005-05-04 18:55:03 +00:00
if (align >= MHLEN)
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
return (NULL);
Unbreak writes of 0 bytes. Zero byte writes happen when only ancillary control data but no payload data is passed. Change m_uiotombuf() to return at least one empty mbuf if the requested length was zero. Add comment to sosend_dgram and sosend_generic(). Diagnoses by: jhb Regression test by: rwatson Pointy hat to. andre
2007-01-22 14:50:28 +00:00
/*
* Give us the full allocation or nothing.
* If len is zero return the smallest empty mbuf.
*/
m = m_getm2(NULL, max(total + align, 1), how, MT_DATA, flags);
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
if (m == NULL)
return (NULL);
m->m_data += align;
/* Fill all mbufs with uio data and update header information. */
for (mb = m; mb != NULL; mb = mb->m_next) {
length = min(M_TRAILINGSPACE(mb), total - progress);
error = uiomove(mtod(mb, void *), length, uio);
if (error) {
m_freem(m);
return (NULL);
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
}
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
mb->m_len = length;
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
progress += length;
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
if (flags & M_PKTHDR)
m->m_pkthdr.len += length;
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
}
Rename m_getm() to m_getm2() and rewrite it to allocate up to page sized mbuf clusters. Add a flags parameter to accept M_PKTHDR and M_EOR mbuf chain flags. Provide compatibility macro for m_getm() calling m_getm2() with M_PKTHDR set. Rewrite m_uiotombuf() to use m_getm2() for mbuf allocation and do the uiomove() in a tight loop over the mbuf chain. Add a flags parameter to accept mbuf flags to be passed to m_getm2(). Adjust all callers for the extra parameter. Sponsored by: TCP/IP Optimization Fundraise 2005 MFC after: 3 month
2006-11-02 17:37:22 +00:00
KASSERT(progress == total, ("%s: progress != total", __func__));
return (m);
Rewrite sendfile's header support so that headers are now sent in the first packet along with data, instead of in their own packet. When serving files of size (packetsize - headersize) or smaller, this will result in one less packet crossing the network. Quick testing with thttpd and http_load has shown a noticeable performance improvement in this case (350 vs 330 fetches per second.) Included in this commit are two support routines, iov_to_uio, and m_uiotombuf; these routines are used by sendfile to construct the header mbuf chain that will be linked to the rest of the data in the socket buffer.
2004-02-01 07:56:44 +00:00
}
add m_align, a function to align any type of mbuf (i.e. it is a superset of M_ALIGN and MH_ALIGN) Reviewed by: several
2005-07-30 01:32:16 +00:00
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
/*
* Copy data from an unmapped mbuf into a uio limited by len if set.
*/
int
m_unmappedtouio(const struct mbuf *m, int m_off, struct uio *uio, int len)
{
vm_page_t pg;
int error, i, off, pglen, pgoff, seglen, segoff;
Mechanically rename MBUF_EXT_PGS_ASSERT() to M_ASSERTEXTPG() to match classical M_ASSERTPKTHDR. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:27:41 +00:00
M_ASSERTEXTPG(m);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
error = 0;
/* Skip over any data removed from the front. */
off = mtod(m, vm_offset_t);
off += m_off;
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
if (m->m_epg_hdrlen != 0) {
if (off >= m->m_epg_hdrlen) {
off -= m->m_epg_hdrlen;
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
} else {
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
seglen = m->m_epg_hdrlen - off;
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
segoff = off;
seglen = min(seglen, len);
off = 0;
len -= seglen;
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
error = uiomove(__DECONST(void *,
&m->m_epg_hdr[segoff]), seglen, uio);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
}
}
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
pgoff = m->m_epg_1st_off;
for (i = 0; i < m->m_epg_npgs && error == 0 && len > 0; i++) {
Step 2.3: Rename mbuf_ext_pg_len() to m_epg_pagelen() that uses mbuf argument. Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 23:52:35 +00:00
pglen = m_epg_pagelen(m, i, pgoff);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
if (off >= pglen) {
off -= pglen;
pgoff = 0;
continue;
}
seglen = pglen - off;
segoff = pgoff + off;
off = 0;
seglen = min(seglen, len);
len -= seglen;
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
pg = PHYS_TO_VM_PAGE(m->m_epg_pa[i]);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
error = uiomove_fromphys(&pg, segoff, seglen, uio);
pgoff = 0;
};
if (len != 0 && error == 0) {
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
KASSERT((off + len) <= m->m_epg_trllen,
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
("off + len > trail (%d + %d > %d, m_off = %d)", off, len,
Step 3: anonymize struct mbuf_ext_pgs and move all its fields into mbuf within m_epg namespace. All edits except the 'struct mbuf' declaration and mb_dupcl() were done mechanically with sed: s/->m_ext_pgs.nrdy/->m_epg_nrdy/g s/->m_ext_pgs.hdr_len/->m_epg_hdrlen/g s/->m_ext_pgs.trail_len/->m_epg_trllen/g s/->m_ext_pgs.first_pg_off/->m_epg_1st_off/g s/->m_ext_pgs.last_pg_len/->m_epg_last_len/g s/->m_ext_pgs.flags/->m_epg_flags/g s/->m_ext_pgs.record_type/->m_epg_record_type/g s/->m_ext_pgs.enc_cnt/->m_epg_enc_cnt/g s/->m_ext_pgs.tls/->m_epg_tls/g s/->m_ext_pgs.so/->m_epg_so/g s/->m_ext_pgs.seqno/->m_epg_seqno/g s/->m_ext_pgs.stailq/->m_epg_stailq/g Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:12:56 +00:00
m->m_epg_trllen, m_off));
Continuation of multi page mbuf redesign from r359919. The following series of patches addresses three things: Now that array of pages is embedded into mbuf, we no longer need separate structure to pass around, so struct mbuf_ext_pgs is an artifact of the first implementation. And struct mbuf_ext_pgs_data is a crutch to accomodate the main idea r359919 with minimal churn. Also, M_EXT of type EXT_PGS are just a synonym of M_NOMAP. The namespace for the newfeature is somewhat inconsistent and sometimes has a lengthy prefixes. In these patches we will gradually bring the namespace to "m_epg" prefix for all mbuf fields and most functions. Step 1 of 4: o Anonymize mbuf_ext_pgs_data, embed in m_ext o Embed mbuf_ext_pgs o Start documenting all this entanglement Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-02 22:39:26 +00:00
error = uiomove(__DECONST(void *, &m->m_epg_trail[off]),
len, uio);
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
}
return (error);
}
Add m_mbuftouio() helper function to copy(out) an arbitrary long mbuf chain into an arbitrary large uio in a single step. It is a functional mirror image of m_uiotombuf(). This function is supposed to be used instead of hand rolled code with the same purpose and to concentrate it into one place for potential further optimization or hardware assistance.
2009-06-22 22:20:38 +00:00
/*
* Copy an mbuf chain into a uio limited by len if set.
*/
int
m_mbuftouio() doesn't modify the mbuf.
2017-03-07 19:00:50 +00:00
m_mbuftouio(struct uio *uio, const struct mbuf *m, int len)
Add m_mbuftouio() helper function to copy(out) an arbitrary long mbuf chain into an arbitrary large uio in a single step. It is a functional mirror image of m_uiotombuf(). This function is supposed to be used instead of hand rolled code with the same purpose and to concentrate it into one place for potential further optimization or hardware assistance.
2009-06-22 22:20:38 +00:00
{
int error, length, total;
int progress = 0;
if (len > 0)
total = min(uio->uio_resid, len);
else
total = uio->uio_resid;
/* Fill the uio with data from the mbufs. */
for (; m != NULL; m = m->m_next) {
length = min(m->m_len, total - progress);
Step 4.1: mechanically rename M_NOMAP to M_EXTPG Reviewed by: gallatin Differential Revision: https://reviews.freebsd.org/D24598
2020-05-03 00:21:11 +00:00
if ((m->m_flags & M_EXTPG) != 0)
Add an external mbuf buffer type that holds multiple unmapped pages. Unmapped mbufs allow sendfile to carry multiple pages of data in a single mbuf, without mapping those pages. It is a requirement for Netflix's in-kernel TLS, and provides a 5-10% CPU savings on heavy web serving workloads when used by sendfile, due to effectively compressing socket buffers by an order of magnitude, and hence reducing cache misses. For this new external mbuf buffer type (EXT_PGS), the ext_buf pointer now points to a struct mbuf_ext_pgs structure instead of a data buffer. This structure contains an array of physical addresses (this reduces cache misses compared to an earlier version that stored an array of vm_page_t pointers). It also stores additional fields needed for in-kernel TLS such as the TLS header and trailer data that are currently unused. To more easily detect these mbufs, the M_NOMAP flag is set in m_flags in addition to M_EXT. Various functions like m_copydata() have been updated to safely access packet contents (using uiomove_fromphys()), to make things like BPF safe. NIC drivers advertise support for unmapped mbufs on transmit via a new IFCAP_NOMAP capability. This capability can be toggled via the new 'nomap' and '-nomap' ifconfig(8) commands. For NIC drivers that only transmit packet contents via DMA and use bus_dma, adding the capability to if_capabilities and if_capenable should be all that is required. If a NIC does not support unmapped mbufs, they are converted to a chain of mapped mbufs (using sf_bufs to provide the mapping) in ip_output or ip6_output. If an unmapped mbuf requires software checksums, it is also converted to a chain of mapped mbufs before computing the checksum. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Discussed with: ae, kp (firewalls) Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:48:33 +00:00
error = m_unmappedtouio(m, 0, uio, length);
else
error = uiomove(mtod(m, void *), length, uio);
Add m_mbuftouio() helper function to copy(out) an arbitrary long mbuf chain into an arbitrary large uio in a single step. It is a functional mirror image of m_uiotombuf(). This function is supposed to be used instead of hand rolled code with the same purpose and to concentrate it into one place for potential further optimization or hardware assistance.
2009-06-22 22:20:38 +00:00
if (error)
return (error);
progress += length;
}
return (0);
}
promote fast ipsec's m_clone routine for public use; it is renamed m_unshare and the caller can now control how mbufs are allocated Reviewed by: andre, luigi, mlaier MFC after: 1 week
2006-03-15 21:11:11 +00:00
/*
* Create a writable copy of the mbuf chain. While doing this
* we compact the chain with a goal of producing a chain with
* at most two mbufs. The second mbuf in this chain is likely
* to be a cluster. The primary purpose of this work is to create
* a writable packet for encryption, compression, etc. The
* secondary goal is to linearize the data so the data can be
* passed to crypto hardware in the most efficient manner possible.
*/
struct mbuf *
m_unshare(struct mbuf *m0, int how)
{
struct mbuf *m, *mprev;
struct mbuf *n, *mfirst, *mlast;
int len, off;
mprev = NULL;
for (m = m0; m != NULL; m = mprev->m_next) {
/*
* Regular mbufs are ignored unless there's a cluster
* in front of it that we can use to coalesce. We do
* the latter mainly so later clusters can be coalesced
* also w/o having to handle them specially (i.e. convert
* mbuf+cluster -> cluster). This optimization is heavily
* influenced by the assumption that we're running over
* Ethernet where MCLBYTES is large enough that the max
* packet size will permit lots of coalescing into a
* single cluster. This in turn permits efficient
* crypto operations, especially when using hardware.
*/
if ((m->m_flags & M_EXT) == 0) {
if (mprev && (mprev->m_flags & M_EXT) &&
m->m_len <= M_TRAILINGSPACE(mprev)) {
/* XXX: this ignores mbuf types */
memcpy(mtod(mprev, caddr_t) + mprev->m_len,
Whitespace only.
2014-05-30 08:22:58 +00:00
mtod(m, caddr_t), m->m_len);
promote fast ipsec's m_clone routine for public use; it is renamed m_unshare and the caller can now control how mbufs are allocated Reviewed by: andre, luigi, mlaier MFC after: 1 week
2006-03-15 21:11:11 +00:00
mprev->m_len += m->m_len;
mprev->m_next = m->m_next; /* unlink from chain */
m_free(m); /* reclaim mbuf */
} else {
mprev = m;
}
continue;
}
/*
* Writable mbufs are left alone (for now).
*/
if (M_WRITABLE(m)) {
mprev = m;
continue;
}
/*
* Not writable, replace with a copy or coalesce with
* the previous mbuf if possible (since we have to copy
* it anyway, we try to reduce the number of mbufs and
* clusters so that future work is easier).
*/
KASSERT(m->m_flags & M_EXT, ("m_flags 0x%x", m->m_flags));
/* NB: we only coalesce into a cluster or larger */
if (mprev != NULL && (mprev->m_flags & M_EXT) &&
m->m_len <= M_TRAILINGSPACE(mprev)) {
/* XXX: this ignores mbuf types */
memcpy(mtod(mprev, caddr_t) + mprev->m_len,
Whitespace only.
2014-05-30 08:22:58 +00:00
mtod(m, caddr_t), m->m_len);
promote fast ipsec's m_clone routine for public use; it is renamed m_unshare and the caller can now control how mbufs are allocated Reviewed by: andre, luigi, mlaier MFC after: 1 week
2006-03-15 21:11:11 +00:00
mprev->m_len += m->m_len;
mprev->m_next = m->m_next; /* unlink from chain */
m_free(m); /* reclaim mbuf */
continue;
}
/*
Contrary to what the deleted comment said, the m_move_pkthdr() will not smash the M_EXT and data pointer, so it is safe to pass an mbuf with external storage procuded by m_getcl() to m_move_pkthdr(). Reviewed by: andre Sponsored by: Nginx, Inc.
2013-03-16 08:55:21 +00:00
* Allocate new space to hold the copy and copy the data.
* We deal with jumbo mbufs (i.e. m_len > MCLBYTES) by
* splitting them into clusters. We could just malloc a
* buffer and make it external but too many device drivers
* don't know how to break up the non-contiguous memory when
promote fast ipsec's m_clone routine for public use; it is renamed m_unshare and the caller can now control how mbufs are allocated Reviewed by: andre, luigi, mlaier MFC after: 1 week
2006-03-15 21:11:11 +00:00
* doing DMA.
*/
Plug leak in m_unshare. m_unshare passes on the source mbuf's flags as-is to m_getcl and this results in a leak if the flags include M_NOFREE. The fix is to clear the bits not listed in M_COPYALL before calling m_getcl. M_RDONLY should probably be filtered out too but that's outside the scope of this fix. Add assertions in the zone_mbuf and zone_pack ctors to catch similar bugs. Update netmap_get_mbuf to not pass M_NOFREE to m_getcl. It's not clear what the original code was trying to do but it's likely incorrect. Updated code is no different functionally but it avoids the newly added assertions. Reviewed by: gnn@ Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D5698
2016-03-26 23:39:53 +00:00
n = m_getcl(how, m->m_type, m->m_flags & M_COPYFLAGS);
Contrary to what the deleted comment said, the m_move_pkthdr() will not smash the M_EXT and data pointer, so it is safe to pass an mbuf with external storage procuded by m_getcl() to m_move_pkthdr(). Reviewed by: andre Sponsored by: Nginx, Inc.
2013-03-16 08:55:21 +00:00
if (n == NULL) {
m_freem(m0);
return (NULL);
}
Fix regression from r248371. We need to copy packet header to new mbuf. Unlike in the pre-r248371 code, assert that M_PKTHDR is set only on a first mbuf. Reported & tested by: Andriy Voskoboinyk <s3erios gmail.com> Sponsored by: Nginx, Inc.
2015-10-07 12:40:00 +00:00
if (m->m_flags & M_PKTHDR) {
KASSERT(mprev == NULL, ("%s: m0 %p, m %p has M_PKTHDR",
__func__, m0, m));
m_move_pkthdr(n, m);
}
promote fast ipsec's m_clone routine for public use; it is renamed m_unshare and the caller can now control how mbufs are allocated Reviewed by: andre, luigi, mlaier MFC after: 1 week
2006-03-15 21:11:11 +00:00
len = m->m_len;
off = 0;
mfirst = n;
mlast = NULL;
for (;;) {
int cc = min(len, MCLBYTES);
memcpy(mtod(n, caddr_t), mtod(m, caddr_t) + off, cc);
n->m_len = cc;
if (mlast != NULL)
mlast->m_next = n;
Whitespace only.
2014-05-30 08:22:58 +00:00
mlast = n;
promote fast ipsec's m_clone routine for public use; it is renamed m_unshare and the caller can now control how mbufs are allocated Reviewed by: andre, luigi, mlaier MFC after: 1 week
2006-03-15 21:11:11 +00:00
#if 0
newipsecstat.ips_clcopied++;
#endif
len -= cc;
if (len <= 0)
break;
off += cc;
Plug leak in m_unshare. m_unshare passes on the source mbuf's flags as-is to m_getcl and this results in a leak if the flags include M_NOFREE. The fix is to clear the bits not listed in M_COPYALL before calling m_getcl. M_RDONLY should probably be filtered out too but that's outside the scope of this fix. Add assertions in the zone_mbuf and zone_pack ctors to catch similar bugs. Update netmap_get_mbuf to not pass M_NOFREE to m_getcl. It's not clear what the original code was trying to do but it's likely incorrect. Updated code is no different functionally but it avoids the newly added assertions. Reviewed by: gnn@ Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D5698
2016-03-26 23:39:53 +00:00
n = m_getcl(how, m->m_type, m->m_flags & M_COPYFLAGS);
promote fast ipsec's m_clone routine for public use; it is renamed m_unshare and the caller can now control how mbufs are allocated Reviewed by: andre, luigi, mlaier MFC after: 1 week
2006-03-15 21:11:11 +00:00
if (n == NULL) {
m_freem(mfirst);
m_freem(m0);
return (NULL);
}
}
Whitespace only.
2014-05-30 08:22:58 +00:00
n->m_next = m->m_next;
promote fast ipsec's m_clone routine for public use; it is renamed m_unshare and the caller can now control how mbufs are allocated Reviewed by: andre, luigi, mlaier MFC after: 1 week
2006-03-15 21:11:11 +00:00
if (mprev == NULL)
m0 = mfirst; /* new head of chain */
else
mprev->m_next = mfirst; /* replace old mbuf */
m_free(m); /* release old mbuf */
mprev = mfirst;
}
return (m0);
}
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
#ifdef MBUF_PROFILING
#define MP_BUCKETS 32 /* don't just change this as things may overflow.*/
struct mbufprofile {
Attempt to make the print types more friendly to other architectures. Prodded by: Max Laier Help from: BMS, jhb
2008-04-30 20:00:30 +00:00
uintmax_t wasted[MP_BUCKETS];
uintmax_t used[MP_BUCKETS];
uintmax_t segments[MP_BUCKETS];
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
} mbprof;
#define MP_MAXDIGITS 21 /* strlen("16,000,000,000,000,000,000") == 21 */
#define MP_NUMLINES 6
#define MP_NUMSPERLINE 16
#define MP_EXTRABYTES 64 /* > strlen("used:\nwasted:\nsegments:\n") */
/* work out max space needed and add a bit of spare space too */
#define MP_MAXLINE ((MP_MAXDIGITS+1) * MP_NUMSPERLINE)
#define MP_BUFSIZE ((MP_MAXLINE * MP_NUMLINES) + 1 + MP_EXTRABYTES)
char mbprofbuf[MP_BUFSIZE];
void
m_profile(struct mbuf *m)
{
int segments = 0;
int used = 0;
int wasted = 0;
Whitespace only.
2014-05-30 08:22:58 +00:00
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
while (m) {
segments++;
used += m->m_len;
if (m->m_flags & M_EXT) {
wasted += MHLEN - sizeof(m->m_ext) +
m->m_ext.ext_size - m->m_len;
} else {
if (m->m_flags & M_PKTHDR)
wasted += MHLEN - m->m_len;
else
wasted += MLEN - m->m_len;
}
m = m->m_next;
}
/* be paranoid.. it helps */
if (segments > MP_BUCKETS - 1)
segments = MP_BUCKETS - 1;
if (used > 100000)
used = 100000;
if (wasted > 100000)
wasted = 100000;
/* store in the appropriate bucket */
/* don't bother locking. if it's slightly off, so what? */
mbprof.segments[segments]++;
mbprof.used[fls(used)]++;
mbprof.wasted[fls(wasted)]++;
}
static void
mbprof_textify(void)
{
int offset;
char *c;
Use ISO C99 integer types in sys/kern where possible. There are only about 100 occurences of the BSD-specific u_int*_t datatypes in sys/kern. The ISO C99 integer types are used here more often.
2010-06-21 09:55:56 +00:00
uint64_t *p;
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
p = &mbprof.wasted[0];
c = mbprofbuf;
Whitespace only.
2014-05-30 08:22:58 +00:00
offset = snprintf(c, MP_MAXLINE + 10,
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
"wasted:\n"
Attempt to make the print types more friendly to other architectures. Prodded by: Max Laier Help from: BMS, jhb
2008-04-30 20:00:30 +00:00
"%ju %ju %ju %ju %ju %ju %ju %ju "
"%ju %ju %ju %ju %ju %ju %ju %ju\n",
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
p[0], p[1], p[2], p[3], p[4], p[5], p[6], p[7],
p[8], p[9], p[10], p[11], p[12], p[13], p[14], p[15]);
#ifdef BIG_ARRAY
p = &mbprof.wasted[16];
c += offset;
Whitespace only.
2014-05-30 08:22:58 +00:00
offset = snprintf(c, MP_MAXLINE,
Attempt to make the print types more friendly to other architectures. Prodded by: Max Laier Help from: BMS, jhb
2008-04-30 20:00:30 +00:00
"%ju %ju %ju %ju %ju %ju %ju %ju "
"%ju %ju %ju %ju %ju %ju %ju %ju\n",
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
p[0], p[1], p[2], p[3], p[4], p[5], p[6], p[7],
p[8], p[9], p[10], p[11], p[12], p[13], p[14], p[15]);
#endif
p = &mbprof.used[0];
c += offset;
Whitespace only.
2014-05-30 08:22:58 +00:00
offset = snprintf(c, MP_MAXLINE + 10,
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
"used:\n"
Attempt to make the print types more friendly to other architectures. Prodded by: Max Laier Help from: BMS, jhb
2008-04-30 20:00:30 +00:00
"%ju %ju %ju %ju %ju %ju %ju %ju "
"%ju %ju %ju %ju %ju %ju %ju %ju\n",
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
p[0], p[1], p[2], p[3], p[4], p[5], p[6], p[7],
p[8], p[9], p[10], p[11], p[12], p[13], p[14], p[15]);
#ifdef BIG_ARRAY
p = &mbprof.used[16];
c += offset;
Whitespace only.
2014-05-30 08:22:58 +00:00
offset = snprintf(c, MP_MAXLINE,
Attempt to make the print types more friendly to other architectures. Prodded by: Max Laier Help from: BMS, jhb
2008-04-30 20:00:30 +00:00
"%ju %ju %ju %ju %ju %ju %ju %ju "
"%ju %ju %ju %ju %ju %ju %ju %ju\n",
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
p[0], p[1], p[2], p[3], p[4], p[5], p[6], p[7],
p[8], p[9], p[10], p[11], p[12], p[13], p[14], p[15]);
#endif
p = &mbprof.segments[0];
c += offset;
Whitespace only.
2014-05-30 08:22:58 +00:00
offset = snprintf(c, MP_MAXLINE + 10,
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
"segments:\n"
Attempt to make the print types more friendly to other architectures. Prodded by: Max Laier Help from: BMS, jhb
2008-04-30 20:00:30 +00:00
"%ju %ju %ju %ju %ju %ju %ju %ju "
"%ju %ju %ju %ju %ju %ju %ju %ju\n",
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
p[0], p[1], p[2], p[3], p[4], p[5], p[6], p[7],
p[8], p[9], p[10], p[11], p[12], p[13], p[14], p[15]);
#ifdef BIG_ARRAY
p = &mbprof.segments[16];
c += offset;
Whitespace only.
2014-05-30 08:22:58 +00:00
offset = snprintf(c, MP_MAXLINE,
Attempt to make the print types more friendly to other architectures. Prodded by: Max Laier Help from: BMS, jhb
2008-04-30 20:00:30 +00:00
"%ju %ju %ju %ju %ju %ju %ju %ju "
"%ju %ju %ju %ju %ju %ju %ju %jju",
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
p[0], p[1], p[2], p[3], p[4], p[5], p[6], p[7],
p[8], p[9], p[10], p[11], p[12], p[13], p[14], p[15]);
#endif
}
static int
mbprof_handler(SYSCTL_HANDLER_ARGS)
{
int error;
mbprof_textify();
error = SYSCTL_OUT(req, mbprofbuf, strlen(mbprofbuf) + 1);
return (error);
}
static int
mbprof_clr_handler(SYSCTL_HANDLER_ARGS)
{
int clear, error;
Whitespace only.
2014-05-30 08:22:58 +00:00
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
clear = 0;
error = sysctl_handle_int(oidp, &clear, 0, req);
if (error || !req->newptr)
return (error);
Whitespace only.
2014-05-30 08:22:58 +00:00
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
if (clear) {
bzero(&mbprof, sizeof(mbprof));
}
Whitespace only.
2014-05-30 08:22:58 +00:00
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
return (error);
}
Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many) r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are still not MPSAFE (or already are but aren’t properly marked). Use it in preparation for a general review of all nodes. This is non-functional change that adds annotations to SYSCTL_NODE and SYSCTL_PROC nodes using one of the soon-to-be-required flags. Mark all obvious cases as MPSAFE. All entries that haven't been marked as MPSAFE before are by default marked as NEEDGIANT Approved by: kib (mentor, blanket) Commented by: kib, gallatin, melifaro Differential Revision: https://reviews.freebsd.org/D23718
2020-02-26 14:26:36 +00:00
SYSCTL_PROC(_kern_ipc, OID_AUTO, mbufprofile,
CTLTYPE_STRING | CTLFLAG_RD | CTLFLAG_NEEDGIANT, NULL, 0,
mbprof_handler, "A",
"mbuf profiling statistics");
SYSCTL_PROC(_kern_ipc, OID_AUTO, mbufprofileclr,
CTLTYPE_INT | CTLFLAG_RW | CTLFLAG_NEEDGIANT, NULL, 0,
mbprof_clr_handler, "I",
"clear mbuf profiling statistics");
Add an option (compiled out by default) to profile outoing packets for a number of mbuf chain related parameters e.g. number of mbufs, wasted space. probably will do with further work later. Reviewed by: various
2008-04-29 21:23:21 +00:00
#endif
Reference in New Issue Copy Permalink