d/freebsd-nq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7244507616
freebsd-nq/sys/kern/kern_sendfile.c

1192 lines
29 KiB
C
Raw Normal View History

- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
/*-
* Copyright (c) 2013-2015 Gleb Smirnoff <glebius@FreeBSD.org>
* Copyright (c) 1998, David Greenman. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
Renumber license clauses in sys/kern to avoid skipping #3
2016-09-15 13:16:20 +00:00
* 3. Neither the name of the University nor the names of its contributors
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/capsicum.h>
#include <sys/kernel.h>
Add support for using unmapped mbufs with sendfile(2). This can be enabled at runtime via the kern.ipc.mb_use_ext_pgs sysctl. It is disabled by default. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:49:35 +00:00
#include <netinet/in.h>
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
#include <sys/lock.h>
#include <sys/mutex.h>
#include <sys/sysproto.h>
#include <sys/malloc.h>
#include <sys/proc.h>
#include <sys/mman.h>
#include <sys/mount.h>
#include <sys/mbuf.h>
#include <sys/protosw.h>
#include <sys/rwlock.h>
#include <sys/sf_buf.h>
#include <sys/socket.h>
#include <sys/socketvar.h>
#include <sys/syscallsubr.h>
#include <sys/sysctl.h>
#include <sys/vnode.h>
#include <net/vnet.h>
#include <security/audit/audit.h>
#include <security/mac/mac_framework.h>
#include <vm/vm.h>
#include <vm/vm_object.h>
#include <vm/vm_pager.h>
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
#define EXT_FLAG_SYNC EXT_FLAG_VENDOR1
#define EXT_FLAG_NOCACHE EXT_FLAG_VENDOR2
Add support for using unmapped mbufs with sendfile(2). This can be enabled at runtime via the kern.ipc.mb_use_ext_pgs sysctl. It is disabled by default. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:49:35 +00:00
#define EXT_FLAG_CACHE_LAST EXT_FLAG_VENDOR3
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
/*
* Structure describing a single sendfile(2) I/O, which may consist of
* several underlying pager I/Os.
*
* The syscall context allocates the structure and initializes 'nios'
* to 1. As sendfile_swapin() runs through pages and starts asynchronous
* paging operations, it increments 'nios'.
*
* Every I/O completion calls sendfile_iodone(), which decrements the 'nios',
* and the syscall also calls sendfile_iodone() after allocating all mbufs,
* linking them and sending to socket. Whoever reaches zero 'nios' is
* responsible to * call pru_ready on the socket, to notify it of readyness
* of the data.
*/
struct sf_io {
volatile u_int nios;
u_int error;
int npages;
Use soref() in sendfile(2) instead fhold() to reference a socket. The problem is that fdrop() requires syscall context, as it may enter sleep in some cases. The reason to use it in the original non-blocking sendfile implementation, was to avoid use of global ACCEPT_LOCK() on every I/O completion. Now in head sorele() no longer requires this lock.
2017-09-13 22:11:05 +00:00
struct socket *so;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
struct mbuf *m;
vm_page_t pa[];
};
/*
* Structure used to track requests with SF_SYNC flag.
*/
struct sendfile_sync {
struct mtx mtx;
struct cv cv;
unsigned count;
};
counter_u64_t sfstat[sizeof(struct sfstat) / sizeof(uint64_t)];
static void
sfstat_init(const void *unused)
{
COUNTER_ARRAY_ALLOC(sfstat, sizeof(struct sfstat) / sizeof(uint64_t),
M_WAITOK);
}
SYSINIT(sfstat, SI_SUB_MBUF, SI_ORDER_FIRST, sfstat_init, NULL);
static int
sfstat_sysctl(SYSCTL_HANDLER_ARGS)
{
struct sfstat s;
COUNTER_ARRAY_COPY(sfstat, &s, sizeof(s) / sizeof(uint64_t));
if (req->newptr)
COUNTER_ARRAY_ZERO(sfstat, sizeof(s) / sizeof(uint64_t));
return (SYSCTL_OUT(req, &s, sizeof(s)));
}
SYSCTL_PROC(_kern_ipc, OID_AUTO, sfstat, CTLTYPE_OPAQUE | CTLFLAG_RW,
NULL, 0, sfstat_sysctl, "I", "sendfile statistics");
/*
* Detach mapped page and release resources back to the system. Called
* by mbuf(9) code when last reference to a page is freed.
*/
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
static void
sendfile_free_page(vm_page_t pg, bool nocache)
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
{
Dequeue wired pages lazily. Previously, wiring a page would cause it to be removed from its page queue. In the common case, unwiring causes it to be enqueued at the tail of that page queue. This change modifies vm_page_wire() to not dequeue the page, thus avoiding the highly contended page queue locks. Instead, vm_page_unwire() takes care of requeuing the page as a single operation, and the page daemon dequeues wired pages as they are encountered during a queue scan to avoid needlessly revisiting them later. For pages in PQ_ACTIVE we do even better, since a requeue is unnecessary. The change improves scalability for some common workloads. For instance, threads wiring pages into the buffer cache no longer need to modify global page queues, and unwiring is usually done by the bufspace thread, so concurrency is not as much of an issue. As another example, many sysctl handlers wire the output buffer to avoid faults on copyout, and since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid modifying the page queue in this case. The change also adds a block comment describing some properties of struct vm_page's reference counters, and the busy lock. Reviewed by: jeff Discussed with: alc, kib MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
bool freed;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
vm_page_lock(pg);
/*
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
* In either case check for the object going away on us. This can
* happen since we don't hold a reference to it. If so, we're
* responsible for freeing the page. In 'noncache' case try to free
* the page, but only if it is cheap to.
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
*/
Dequeue wired pages lazily. Previously, wiring a page would cause it to be removed from its page queue. In the common case, unwiring causes it to be enqueued at the tail of that page queue. This change modifies vm_page_wire() to not dequeue the page, thus avoiding the highly contended page queue locks. Instead, vm_page_unwire() takes care of requeuing the page as a single operation, and the page daemon dequeues wired pages as they are encountered during a queue scan to avoid needlessly revisiting them later. For pages in PQ_ACTIVE we do even better, since a requeue is unnecessary. The change improves scalability for some common workloads. For instance, threads wiring pages into the buffer cache no longer need to modify global page queues, and unwiring is usually done by the bufspace thread, so concurrency is not as much of an issue. As another example, many sysctl handlers wire the output buffer to avoid faults on copyout, and since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid modifying the page queue in this case. The change also adds a block comment describing some properties of struct vm_page's reference counters, and the busy lock. Reviewed by: jeff Discussed with: alc, kib MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
if (vm_page_unwire_noq(pg)) {
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
vm_object_t obj;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
if ((obj = pg->object) == NULL)
vm_page_free(pg);
Dequeue wired pages lazily. Previously, wiring a page would cause it to be removed from its page queue. In the common case, unwiring causes it to be enqueued at the tail of that page queue. This change modifies vm_page_wire() to not dequeue the page, thus avoiding the highly contended page queue locks. Instead, vm_page_unwire() takes care of requeuing the page as a single operation, and the page daemon dequeues wired pages as they are encountered during a queue scan to avoid needlessly revisiting them later. For pages in PQ_ACTIVE we do even better, since a requeue is unnecessary. The change improves scalability for some common workloads. For instance, threads wiring pages into the buffer cache no longer need to modify global page queues, and unwiring is usually done by the bufspace thread, so concurrency is not as much of an issue. As another example, many sysctl handlers wire the output buffer to avoid faults on copyout, and since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid modifying the page queue in this case. The change also adds a block comment describing some properties of struct vm_page's reference counters, and the busy lock. Reviewed by: jeff Discussed with: alc, kib MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
else {
freed = false;
if (nocache && !vm_page_xbusied(pg) &&
VM_OBJECT_TRYWLOCK(obj)) {
Address two problems with sendfile(..., SF_NOCACHE) and apply one "optimization". First, sendfile(..., SF_NOCACHE) frees pages without checking whether those pages are mapped. This can leave the system with mappings to free or repurposed pages. Second, a page can be busied between the time of the current busy test and acquiring the object lock. Essentially, the test performed before the object lock is acquired can only be regarded as an optimization to short-circuit further work on the page. It cannot, however, be relied upon to prove that it is safe to free the page. Third, when sendfile(..., SF_NOCACHE) was originally implemented, vm_page_deactivate_noreuse() did not yet exist. Use vm_page_deactivate_noreuse() instead of vm_page_deactivate(), because it comes closer to freeing the page. In collaboration with: glebius Discussed with: gallatin, kib, markj X-MFC after: r324448
2017-10-13 16:31:50 +00:00
/* Only free unmapped pages. */
if (obj->ref_count == 0 ||
!pmap_page_is_mapped(pg))
/*
* The busy test before the object is
* locked cannot be relied upon.
*/
freed = vm_page_try_to_free(pg);
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
VM_OBJECT_WUNLOCK(obj);
Dequeue wired pages lazily. Previously, wiring a page would cause it to be removed from its page queue. In the common case, unwiring causes it to be enqueued at the tail of that page queue. This change modifies vm_page_wire() to not dequeue the page, thus avoiding the highly contended page queue locks. Instead, vm_page_unwire() takes care of requeuing the page as a single operation, and the page daemon dequeues wired pages as they are encountered during a queue scan to avoid needlessly revisiting them later. For pages in PQ_ACTIVE we do even better, since a requeue is unnecessary. The change improves scalability for some common workloads. For instance, threads wiring pages into the buffer cache no longer need to modify global page queues, and unwiring is usually done by the bufspace thread, so concurrency is not as much of an issue. As another example, many sysctl handlers wire the output buffer to avoid faults on copyout, and since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid modifying the page queue in this case. The change also adds a block comment describing some properties of struct vm_page's reference counters, and the busy lock. Reviewed by: jeff Discussed with: alc, kib MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
}
if (!freed) {
/*
* If we were asked to not cache the page, place
* it near the head of the inactive queue so
* that it is reclaimed sooner. Otherwise,
* maintain LRU.
*/
if (nocache)
Address two problems with sendfile(..., SF_NOCACHE) and apply one "optimization". First, sendfile(..., SF_NOCACHE) frees pages without checking whether those pages are mapped. This can leave the system with mappings to free or repurposed pages. Second, a page can be busied between the time of the current busy test and acquiring the object lock. Essentially, the test performed before the object lock is acquired can only be regarded as an optimization to short-circuit further work on the page. It cannot, however, be relied upon to prove that it is safe to free the page. Third, when sendfile(..., SF_NOCACHE) was originally implemented, vm_page_deactivate_noreuse() did not yet exist. Use vm_page_deactivate_noreuse() instead of vm_page_deactivate(), because it comes closer to freeing the page. In collaboration with: glebius Discussed with: gallatin, kib, markj X-MFC after: r324448
2017-10-13 16:31:50 +00:00
vm_page_deactivate_noreuse(pg);
Fix some races introduced in r332974. With r332974, when performing a synchronized access of a page's "queue" field, one must first check whether the page is logically dequeued. If so, then the page lock does not prevent the page from being removed from its page queue. Intoduce vm_page_queue(), which returns the page's logical queue index. In some cases, direct access to the "queue" field is still required, but such accesses should be confined to sys/vm. Reported and tested by: pho Reviewed by: kib Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D15280
2018-05-04 17:17:30 +00:00
else if (vm_page_active(pg))
Dequeue wired pages lazily. Previously, wiring a page would cause it to be removed from its page queue. In the common case, unwiring causes it to be enqueued at the tail of that page queue. This change modifies vm_page_wire() to not dequeue the page, thus avoiding the highly contended page queue locks. Instead, vm_page_unwire() takes care of requeuing the page as a single operation, and the page daemon dequeues wired pages as they are encountered during a queue scan to avoid needlessly revisiting them later. For pages in PQ_ACTIVE we do even better, since a requeue is unnecessary. The change improves scalability for some common workloads. For instance, threads wiring pages into the buffer cache no longer need to modify global page queues, and unwiring is usually done by the bufspace thread, so concurrency is not as much of an issue. As another example, many sysctl handlers wire the output buffer to avoid faults on copyout, and since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid modifying the page queue in this case. The change also adds a block comment describing some properties of struct vm_page's reference counters, and the busy lock. Reviewed by: jeff Discussed with: alc, kib MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
vm_page_reference(pg);
else
Have vm_page_{deactivate,launder}() requeue already-queued pages. In many cases the page is not enqueued so the change will have no effect. However, the change is needed to support an optimization in the fault handler and in some cases (sendfile, the buffer cache) it was being emulated by the caller anyway. Reviewed by: alc Tested by: pho MFC after: 2 weeks X-Differential Revision: https://reviews.freebsd.org/D14625
2018-03-18 16:40:56 +00:00
vm_page_deactivate(pg);
Dequeue wired pages lazily. Previously, wiring a page would cause it to be removed from its page queue. In the common case, unwiring causes it to be enqueued at the tail of that page queue. This change modifies vm_page_wire() to not dequeue the page, thus avoiding the highly contended page queue locks. Instead, vm_page_unwire() takes care of requeuing the page as a single operation, and the page daemon dequeues wired pages as they are encountered during a queue scan to avoid needlessly revisiting them later. For pages in PQ_ACTIVE we do even better, since a requeue is unnecessary. The change improves scalability for some common workloads. For instance, threads wiring pages into the buffer cache no longer need to modify global page queues, and unwiring is usually done by the bufspace thread, so concurrency is not as much of an issue. As another example, many sysctl handlers wire the output buffer to avoid faults on copyout, and since the buffer is likely to be in PQ_ACTIVE, we now entirely avoid modifying the page queue in this case. The change also adds a block comment describing some properties of struct vm_page's reference counters, and the busy lock. Reviewed by: jeff Discussed with: alc, kib MFC after: 1 month Differential Revision: https://reviews.freebsd.org/D11943
2018-02-07 16:57:10 +00:00
}
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
}
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
vm_page_unlock(pg);
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
}
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
static void
sendfile_free_mext(struct mbuf *m)
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
{
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
struct sf_buf *sf;
vm_page_t pg;
bool nocache;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
KASSERT(m->m_flags & M_EXT && m->m_ext.ext_type == EXT_SFBUF,
("%s: m %p !M_EXT or !EXT_SFBUF", __func__, m));
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
sf = m->m_ext.ext_arg1;
pg = sf_buf_page(sf);
nocache = m->m_ext.ext_flags & EXT_FLAG_NOCACHE;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
sf_buf_free(sf);
sendfile_free_page(pg, nocache);
if (m->m_ext.ext_flags & EXT_FLAG_SYNC) {
struct sendfile_sync *sfs = m->m_ext.ext_arg2;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
mtx_lock(&sfs->mtx);
KASSERT(sfs->count > 0, ("Sendfile sync botchup count == 0"));
if (--sfs->count == 0)
cv_signal(&sfs->cv);
mtx_unlock(&sfs->mtx);
}
}
Add support for using unmapped mbufs with sendfile(2). This can be enabled at runtime via the kern.ipc.mb_use_ext_pgs sysctl. It is disabled by default. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:49:35 +00:00
static void
sendfile_free_mext_pg(struct mbuf *m)
{
struct mbuf_ext_pgs *ext_pgs;
vm_page_t pg;
int i;
bool nocache, cache_last;
KASSERT(m->m_flags & M_EXT && m->m_ext.ext_type == EXT_PGS,
("%s: m %p !M_EXT or !EXT_PGS", __func__, m));
nocache = m->m_ext.ext_flags & EXT_FLAG_NOCACHE;
cache_last = m->m_ext.ext_flags & EXT_FLAG_CACHE_LAST;
ext_pgs = m->m_ext.ext_pgs;
for (i = 0; i < ext_pgs->npgs; i++) {
if (cache_last && i == ext_pgs->npgs - 1)
nocache = false;
pg = PHYS_TO_VM_PAGE(ext_pgs->pa[i]);
sendfile_free_page(pg, nocache);
}
if (m->m_ext.ext_flags & EXT_FLAG_SYNC) {
struct sendfile_sync *sfs = m->m_ext.ext_arg2;
mtx_lock(&sfs->mtx);
KASSERT(sfs->count > 0, ("Sendfile sync botchup count == 0"));
if (--sfs->count == 0)
cv_signal(&sfs->cv);
mtx_unlock(&sfs->mtx);
}
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
/*
* Helper function to calculate how much data to put into page i of n.
* Only first and last pages are special.
*/
static inline off_t
xfsize(int i, int n, off_t off, off_t len)
{
if (i == 0)
return (omin(PAGE_SIZE - (off & PAGE_MASK), len));
if (i == n - 1 && ((off + len) & PAGE_MASK) > 0)
return ((off + len) & PAGE_MASK);
return (PAGE_SIZE);
}
/*
* Helper function to get offset within object for i page.
*/
The data type returned by vmoff() is too narrow in its range. This could break the transmission of files longer than 4 GB on 32-bit architectures. Reviewed by: glebius, kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D10019
2017-06-03 16:19:33 +00:00
static inline vm_ooffset_t
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
vmoff(int i, off_t off)
{
if (i == 0)
The data type returned by vmoff() is too narrow in its range. This could break the transmission of files longer than 4 GB on 32-bit architectures. Reviewed by: glebius, kib MFC after: 1 week Differential Revision: https://reviews.freebsd.org/D10019
2017-06-03 16:19:33 +00:00
return ((vm_ooffset_t)off);
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
return (trunc_page(off + i * PAGE_SIZE));
}
/*
* Helper function used when allocation of a page or sf_buf failed.
* Pretend as if we don't have enough space, subtract xfsize() of
* all pages that failed.
*/
static inline void
fixspace(int old, int new, off_t off, int *space)
{
KASSERT(old > new, ("%s: old %d new %d", __func__, old, new));
/* Subtract last one. */
*space -= xfsize(old - 1, old, off, *space);
old--;
if (new == old)
/* There was only one page. */
return;
/* Subtract first one. */
if (new == 0) {
*space -= xfsize(0, old, off, *space);
new++;
}
/* Rest of pages are full sized. */
*space -= (old - new) * PAGE_SIZE;
KASSERT(*space >= 0, ("%s: space went backwards", __func__));
}
/*
* I/O completion callback.
*/
static void
sendfile_iodone(void *arg, vm_page_t *pg, int count, int error)
{
struct sf_io *sfio = arg;
Use soref() in sendfile(2) instead fhold() to reference a socket. The problem is that fdrop() requires syscall context, as it may enter sleep in some cases. The reason to use it in the original non-blocking sendfile implementation, was to avoid use of global ACCEPT_LOCK() on every I/O completion. Now in head sorele() no longer requires this lock.
2017-09-13 22:11:05 +00:00
struct socket *so = sfio->so;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
for (int i = 0; i < count; i++)
Use bogus_page to properly reduce number of I/Os in sendfile(2). The new sendfile_swapin() loop works this way: - Find first invalid page in the request. - Do vm_pager_has_page() and get count of pages, that can be taken in single I/O. - Trim valid pages from the end of the request. - Cycle through the request and substitute to bogus_page all valid pages that are in the middle of the request. - After I/O launched (pager copies array of pages into buf(9), it is important to restore proper page pointers with help vm_page_lookup(). Count bogus pages used and report them in sendfile stats.
2016-11-17 21:02:55 +00:00
if (pg[i] != bogus_page)
vm_page_xunbusy(pg[i]);
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
if (error)
sfio->error = error;
if (!refcount_release(&sfio->nios))
return;
In sendfile_iodone() both pru_abort and sorele need to be executed with proper VNET context set. Reported by: sbruno MFC after: 2 weeks
2018-01-05 20:21:46 +00:00
CURVNET_SET(so->so_vnet);
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
if (sfio->error) {
/*
* I/O operation failed. The state of data in the socket
* is now inconsistent, and all what we can do is to tear
* it down. Protocol abort method would tear down protocol
* state, free all ready mbufs and detach not ready ones.
* We will free the mbufs corresponding to this I/O manually.
*
* The socket would be marked with EIO and made available
* for read, so that application receives EIO on next
* syscall and eventually closes the socket.
*/
so->so_proto->pr_usrreqs->pru_abort(so);
so->so_error = EIO;
Add support for using unmapped mbufs with sendfile(2). This can be enabled at runtime via the kern.ipc.mb_use_ext_pgs sysctl. It is disabled by default. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:49:35 +00:00
mb_free_notready(sfio->m, sfio->npages);
In sendfile_iodone() both pru_abort and sorele need to be executed with proper VNET context set. Reported by: sbruno MFC after: 2 weeks
2018-01-05 20:21:46 +00:00
} else
Trim an extra space.
2019-06-11 22:06:05 +00:00
(void)(so->so_proto->pr_usrreqs->pru_ready)(so, sfio->m,
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
sfio->npages);
Use soref() in sendfile(2) instead fhold() to reference a socket. The problem is that fdrop() requires syscall context, as it may enter sleep in some cases. The reason to use it in the original non-blocking sendfile implementation, was to avoid use of global ACCEPT_LOCK() on every I/O completion. Now in head sorele() no longer requires this lock.
2017-09-13 22:11:05 +00:00
SOCK_LOCK(so);
sorele(so);
In sendfile_iodone() both pru_abort and sorele need to be executed with proper VNET context set. Reported by: sbruno MFC after: 2 weeks
2018-01-05 20:21:46 +00:00
CURVNET_RESTORE();
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
free(sfio, M_TEMP);
}
/*
* Iterate through pages vector and request paging for non-valid pages.
*/
static int
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
sendfile_swapin(vm_object_t obj, struct sf_io *sfio, int *nios, off_t off,
off_t len, int npages, int rhpages, int flags)
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
{
vm_page_t *pa = sfio->pa;
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
int grabbed;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
*nios = 0;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
flags = (flags & SF_NODISKIO) ? VM_ALLOC_NOWAIT : 0;
/*
* First grab all the pages and wire them. Note that we grab
* only required pages. Readahead pages are dealt with later.
*/
VM_OBJECT_WLOCK(obj);
Have sendfile_swapin() use vm_page_grab_pages(). Reviewed by: alc, kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D11942
2017-08-11 16:32:24 +00:00
grabbed = vm_page_grab_pages(obj, OFF_TO_IDX(off),
VM_ALLOC_NORMAL | VM_ALLOC_WIRED | flags, pa, npages);
if (grabbed < npages) {
for (int i = grabbed; i < npages; i++)
pa[i] = NULL;
npages = grabbed;
rhpages = 0;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
}
for (int i = 0; i < npages;) {
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
int j, a, count, rv;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
/* Skip valid pages. */
if (vm_page_is_valid(pa[i], vmoff(i, off) & PAGE_MASK,
xfsize(i, npages, off, len))) {
vm_page_xunbusy(pa[i]);
SFSTAT_INC(sf_pages_valid);
i++;
continue;
}
/*
Use bogus_page to properly reduce number of I/Os in sendfile(2). The new sendfile_swapin() loop works this way: - Find first invalid page in the request. - Do vm_pager_has_page() and get count of pages, that can be taken in single I/O. - Trim valid pages from the end of the request. - Cycle through the request and substitute to bogus_page all valid pages that are in the middle of the request. - After I/O launched (pager copies array of pages into buf(9), it is important to restore proper page pointers with help vm_page_lookup(). Count bogus pages used and report them in sendfile stats.
2016-11-17 21:02:55 +00:00
* Next page is invalid. Check if it belongs to pager. It
* may not be there, which is a regular situation for shmem
* pager. For vnode pager this happens only in case of
* a sparse file.
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
*
* Important feature of vm_pager_has_page() is the hint
* stored in 'a', about how many pages we can pagein after
* this page in a single I/O.
*/
Use bogus_page to properly reduce number of I/Os in sendfile(2). The new sendfile_swapin() loop works this way: - Find first invalid page in the request. - Do vm_pager_has_page() and get count of pages, that can be taken in single I/O. - Trim valid pages from the end of the request. - Cycle through the request and substitute to bogus_page all valid pages that are in the middle of the request. - After I/O launched (pager copies array of pages into buf(9), it is important to restore proper page pointers with help vm_page_lookup(). Count bogus pages used and report them in sendfile stats.
2016-11-17 21:02:55 +00:00
if (!vm_pager_has_page(obj, OFF_TO_IDX(vmoff(i, off)), NULL,
&a)) {
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
pmap_zero_page(pa[i]);
pa[i]->valid = VM_PAGE_BITS_ALL;
An invalid page can't be dirty. Reviewed by: kib MFC after: 1 week
2017-08-11 16:27:54 +00:00
MPASS(pa[i]->dirty == 0);
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
vm_page_xunbusy(pa[i]);
i++;
continue;
Use bogus_page to properly reduce number of I/Os in sendfile(2). The new sendfile_swapin() loop works this way: - Find first invalid page in the request. - Do vm_pager_has_page() and get count of pages, that can be taken in single I/O. - Trim valid pages from the end of the request. - Cycle through the request and substitute to bogus_page all valid pages that are in the middle of the request. - After I/O launched (pager copies array of pages into buf(9), it is important to restore proper page pointers with help vm_page_lookup(). Count bogus pages used and report them in sendfile stats.
2016-11-17 21:02:55 +00:00
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
/*
* We want to pagein as many pages as possible, limited only
* by the 'a' hint and actual request.
*/
count = min(a + 1, npages - i);
Use bogus_page to properly reduce number of I/Os in sendfile(2). The new sendfile_swapin() loop works this way: - Find first invalid page in the request. - Do vm_pager_has_page() and get count of pages, that can be taken in single I/O. - Trim valid pages from the end of the request. - Cycle through the request and substitute to bogus_page all valid pages that are in the middle of the request. - After I/O launched (pager copies array of pages into buf(9), it is important to restore proper page pointers with help vm_page_lookup(). Count bogus pages used and report them in sendfile stats.
2016-11-17 21:02:55 +00:00
/*
* We should not pagein into a valid page, thus we first trim
* any valid pages off the end of request, and substitute
* to bogus_page those, that are in the middle.
*/
for (j = i + count - 1; j > i; j--) {
if (vm_page_is_valid(pa[j], vmoff(j, off) & PAGE_MASK,
xfsize(j, npages, off, len))) {
count--;
rhpages = 0;
} else
break;
}
for (j = i + 1; j < i + count - 1; j++)
if (vm_page_is_valid(pa[j], vmoff(j, off) & PAGE_MASK,
xfsize(j, npages, off, len))) {
vm_page_xunbusy(pa[j]);
SFSTAT_INC(sf_pages_valid);
SFSTAT_INC(sf_pages_bogus);
pa[j] = bogus_page;
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
refcount_acquire(&sfio->nios);
rv = vm_pager_get_pages_async(obj, pa + i, count, NULL,
i + count == npages ? &rhpages : NULL,
&sendfile_iodone, sfio);
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
if (rv != VM_PAGER_OK) {
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error This is a partial merge of 350144 from projects/fuse2 PR: 236466 Reviewed by: markj MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21095
2019-07-29 20:50:26 +00:00
for (j = i; j < i + count; j++) {
if (pa[j] != bogus_page) {
vm_page_lock(pa[j]);
vm_page_unwire(pa[j], PQ_INACTIVE);
vm_page_unlock(pa[j]);
}
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
}
VM_OBJECT_WUNLOCK(obj);
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error This is a partial merge of 350144 from projects/fuse2 PR: 236466 Reviewed by: markj MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21095
2019-07-29 20:50:26 +00:00
return (EIO);
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
KASSERT(rv == VM_PAGER_OK, ("%s: pager fail obj %p page %p",
__func__, obj, pa[i]));
SFSTAT_INC(sf_iocnt);
SFSTAT_ADD(sf_pages_read, count);
if (i + count == npages)
SFSTAT_ADD(sf_rhpages_read, rhpages);
Use bogus_page to properly reduce number of I/Os in sendfile(2). The new sendfile_swapin() loop works this way: - Find first invalid page in the request. - Do vm_pager_has_page() and get count of pages, that can be taken in single I/O. - Trim valid pages from the end of the request. - Cycle through the request and substitute to bogus_page all valid pages that are in the middle of the request. - After I/O launched (pager copies array of pages into buf(9), it is important to restore proper page pointers with help vm_page_lookup(). Count bogus pages used and report them in sendfile stats.
2016-11-17 21:02:55 +00:00
/*
* Restore the valid page pointers. They are already
* unbusied, but still wired.
*/
for (j = i; j < i + count; j++)
if (pa[j] == bogus_page) {
pa[j] = vm_page_lookup(obj,
OFF_TO_IDX(vmoff(j, off)));
KASSERT(pa[j], ("%s: page %p[%d] disappeared",
__func__, pa, j));
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
i += count;
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
(*nios)++;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
}
VM_OBJECT_WUNLOCK(obj);
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
if (*nios == 0 && npages != 0)
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
SFSTAT_INC(sf_noiocnt);
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
return (0);
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
}
static int
sendfile_getobj(struct thread *td, struct file *fp, vm_object_t *obj_res,
struct vnode **vp_res, struct shmfd **shmfd_res, off_t *obj_size,
int *bsize)
{
struct vattr va;
vm_object_t obj;
struct vnode *vp;
struct shmfd *shmfd;
int error;
vp = *vp_res = NULL;
obj = NULL;
shmfd = *shmfd_res = NULL;
*bsize = 0;
/*
* The file descriptor must be a regular file and have a
* backing VM object.
*/
if (fp->f_type == DTYPE_VNODE) {
vp = fp->f_vnode;
vn_lock(vp, LK_SHARED | LK_RETRY);
if (vp->v_type != VREG) {
error = EINVAL;
goto out;
}
*bsize = vp->v_mount->mnt_stat.f_iosize;
error = VOP_GETATTR(vp, &va, td->td_ucred);
if (error != 0)
goto out;
*obj_size = va.va_size;
obj = vp->v_object;
if (obj == NULL) {
error = EINVAL;
goto out;
}
} else if (fp->f_type == DTYPE_SHM) {
error = 0;
shmfd = fp->f_data;
obj = shmfd->shm_object;
*obj_size = shmfd->shm_size;
} else {
error = EINVAL;
goto out;
}
VM_OBJECT_WLOCK(obj);
if ((obj->flags & OBJ_DEAD) != 0) {
VM_OBJECT_WUNLOCK(obj);
error = EBADF;
goto out;
}
/*
* Temporarily increase the backing VM object's reference
* count so that a forced reclamation of its vnode does not
* immediately destroy it.
*/
vm_object_reference_locked(obj);
VM_OBJECT_WUNLOCK(obj);
*obj_res = obj;
*vp_res = vp;
*shmfd_res = shmfd;
out:
if (vp != NULL)
VOP_UNLOCK(vp, 0);
return (error);
}
static int
sendfile_getsock(struct thread *td, int s, struct file **sock_fp,
struct socket **so)
{
int error;
*sock_fp = NULL;
*so = NULL;
/*
* The socket must be a stream socket and connected.
*/
Eliminate the overhead of gratuitous repeated reinitialization of cap_rights - Add macros to allow preinitialization of cap_rights_t. - Convert most commonly used code paths to use preinitialized cap_rights_t. A 3.6% speedup in fstat was measured with this change. Reported by: mjg Reviewed by: oshogbo Approved by: sbruno MFC after: 1 month
2018-05-09 18:47:24 +00:00
error = getsock_cap(td, s, &cap_send_rights,
capsicum: propagate rights on accept(2) Descriptor returned by accept(2) should inherits capabilities rights from the listening socket. PR: 201052 Reviewed by: emaste, jonathan Discussed with: many Differential Revision: https://reviews.freebsd.org/D7724
2016-09-22 09:58:46 +00:00
sock_fp, NULL, NULL);
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
if (error != 0)
return (error);
*so = (*sock_fp)->f_data;
if ((*so)->so_type != SOCK_STREAM)
return (EINVAL);
Plug sendfile(2) on a listening socket with proper error code. Reported by: ngie Reviewed by: ngie Approved by: re (delphij)
2018-10-16 15:57:16 +00:00
if (SOLISTENING(*so))
return (ENOTCONN);
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
return (0);
}
int
vn_sendfile(struct file *fp, int sockfd, struct uio *hdr_uio,
struct uio *trl_uio, off_t offset, size_t nbytes, off_t *sent, int flags,
The sendfile(2) allows to send extra data from userspace before the file data (headers). Historically the size of the headers was not checked against the socket buffer space. Application could easily overcommit the socket buffer space. With the new sendfile (r293439) the problem remained, but a KASSERT was inserted that checked that amount of data written to the socket matches its space. In case when size of headers is bigger that socket space, KASSERT fires. Without INVARIANTS the new sendfile won't panic, but would report incorrect amount of bytes sent. o With this change, the headers copyin is moved down into the cycle, after the sbspace() check. The uio size is trimmed by socket space there, which fixes the overcommit problem and its consequences. o The compatibility handling for FreeBSD 4 sendfile headers API is pushed up the stack to syscall wrappers. This required a copy and paste of the code, but in turn this allowed to remove extra stack carried parameter from fo_sendfile_t, and embrace entire compat code into #ifdef. If in future we got more fo_sendfile_t function, the copy and paste level would even reduce. Reviewed by: emax, gallatin, Maxim Dounin <mdounin mdounin.ru> Tested by: Vitalij Satanivskij <satan ukr.net> Sponsored by: Netflix
2016-03-29 19:57:11 +00:00
struct thread *td)
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
{
struct file *sock_fp;
struct vnode *vp;
struct vm_object *obj;
struct socket *so;
Add support for using unmapped mbufs with sendfile(2). This can be enabled at runtime via the kern.ipc.mb_use_ext_pgs sysctl. It is disabled by default. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:49:35 +00:00
struct mbuf_ext_pgs *ext_pgs;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
struct mbuf *m, *mh, *mhtail;
struct sf_buf *sf;
struct shmfd *shmfd;
struct sendfile_sync *sfs;
struct vattr va;
off_t off, sbytes, rem, obj_size;
Add support for using unmapped mbufs with sendfile(2). This can be enabled at runtime via the kern.ipc.mb_use_ext_pgs sysctl. It is disabled by default. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:49:35 +00:00
int bsize, error, ext_pgs_idx, hdrlen, max_pgs, softerr;
bool use_ext_pgs;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
obj = NULL;
so = NULL;
m = mh = NULL;
sfs = NULL;
The sendfile(2) allows to send extra data from userspace before the file data (headers). Historically the size of the headers was not checked against the socket buffer space. Application could easily overcommit the socket buffer space. With the new sendfile (r293439) the problem remained, but a KASSERT was inserted that checked that amount of data written to the socket matches its space. In case when size of headers is bigger that socket space, KASSERT fires. Without INVARIANTS the new sendfile won't panic, but would report incorrect amount of bytes sent. o With this change, the headers copyin is moved down into the cycle, after the sbspace() check. The uio size is trimmed by socket space there, which fixes the overcommit problem and its consequences. o The compatibility handling for FreeBSD 4 sendfile headers API is pushed up the stack to syscall wrappers. This required a copy and paste of the code, but in turn this allowed to remove extra stack carried parameter from fo_sendfile_t, and embrace entire compat code into #ifdef. If in future we got more fo_sendfile_t function, the copy and paste level would even reduce. Reviewed by: emax, gallatin, Maxim Dounin <mdounin mdounin.ru> Tested by: Vitalij Satanivskij <satan ukr.net> Sponsored by: Netflix
2016-03-29 19:57:11 +00:00
hdrlen = sbytes = 0;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
softerr = 0;
Add support for using unmapped mbufs with sendfile(2). This can be enabled at runtime via the kern.ipc.mb_use_ext_pgs sysctl. It is disabled by default. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:49:35 +00:00
use_ext_pgs = false;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
error = sendfile_getobj(td, fp, &obj, &vp, &shmfd, &obj_size, &bsize);
if (error != 0)
return (error);
error = sendfile_getsock(td, sockfd, &sock_fp, &so);
if (error != 0)
goto out;
#ifdef MAC
error = mac_socket_check_send(td->td_ucred, so);
if (error != 0)
goto out;
#endif
SFSTAT_INC(sf_syscalls);
SFSTAT_ADD(sf_rhpages_requested, SF_READAHEAD(flags));
if (flags & SF_SYNC) {
sfs = malloc(sizeof *sfs, M_TEMP, M_WAITOK | M_ZERO);
mtx_init(&sfs->mtx, "sendfile", NULL, MTX_DEF);
cv_init(&sfs->cv, "sendfile");
}
rem = nbytes ? omin(nbytes, obj_size - offset) : obj_size - offset;
/*
* Protect against multiple writers to the socket.
*
* XXXRW: Historically this has assumed non-interruptibility, so now
* we implement that, but possibly shouldn't.
*/
(void)sblock(&so->so_snd, SBL_WAIT | SBL_NOINTR);
/*
* Loop through the pages of the file, starting with the requested
* offset. Get a file page (do I/O if necessary), map the file page
* into an sf_buf, attach an mbuf header to the sf_buf, and queue
* it on the socket.
* This is done in two loops. The inner loop turns as many pages
* as it can, up to available socket buffer space, without blocking
* into mbufs to have it bulk delivered into the socket send buffer.
* The outer loop checks the state and available space of the socket
* and takes care of the overall progress.
*/
for (off = offset; rem > 0; ) {
struct sf_io *sfio;
vm_page_t *pa;
struct mbuf *mtail;
int nios, space, npages, rhpages;
mtail = NULL;
/*
* Check the socket state for ongoing connection,
* no errors and space in socket buffer.
* If space is low allow for the remainder of the
* file to be processed if it fits the socket buffer.
* Otherwise block in waiting for sufficient space
* to proceed, or if the socket is nonblocking, return
* to userland with EAGAIN while reporting how far
* we've come.
* We wait until the socket buffer has significant free
* space to do bulk sends. This makes good use of file
* system read ahead and allows packet segmentation
* offloading hardware to take over lots of work. If
* we were not careful here we would send off only one
* sfbuf at a time.
*/
SOCKBUF_LOCK(&so->so_snd);
if (so->so_snd.sb_lowat < so->so_snd.sb_hiwat / 2)
so->so_snd.sb_lowat = so->so_snd.sb_hiwat / 2;
retry_space:
if (so->so_snd.sb_state & SBS_CANTSENDMORE) {
error = EPIPE;
SOCKBUF_UNLOCK(&so->so_snd);
goto done;
} else if (so->so_error) {
error = so->so_error;
so->so_error = 0;
SOCKBUF_UNLOCK(&so->so_snd);
goto done;
}
match sendfile() error handling to send(). Sendfile() should match the error checking order of send() which is currently: SBS_CANTSENDMORE so_error SS_ISCONNECTED Submitted by: Jason Eggleston <jason@eggnet.com> Reviewed by: glebius MFC after: 2 weeks Sponsored by: Limelight Networks Differential Revision: https://reviews.freebsd.org/D12633
2017-10-10 22:21:05 +00:00
if ((so->so_state & SS_ISCONNECTED) == 0) {
SOCKBUF_UNLOCK(&so->so_snd);
error = ENOTCONN;
goto done;
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
space = sbspace(&so->so_snd);
if (space < rem &&
(space <= 0 ||
space < so->so_snd.sb_lowat)) {
if (so->so_state & SS_NBIO) {
SOCKBUF_UNLOCK(&so->so_snd);
error = EAGAIN;
goto done;
}
/*
* sbwait drops the lock while sleeping.
* When we loop back to retry_space the
* state may have changed and we retest
* for it.
*/
error = sbwait(&so->so_snd);
/*
* An error from sbwait usually indicates that we've
* been interrupted by a signal. If we've sent anything
* then return bytes sent, otherwise return the error.
*/
if (error != 0) {
SOCKBUF_UNLOCK(&so->so_snd);
goto done;
}
goto retry_space;
}
SOCKBUF_UNLOCK(&so->so_snd);
/*
The sendfile(2) allows to send extra data from userspace before the file data (headers). Historically the size of the headers was not checked against the socket buffer space. Application could easily overcommit the socket buffer space. With the new sendfile (r293439) the problem remained, but a KASSERT was inserted that checked that amount of data written to the socket matches its space. In case when size of headers is bigger that socket space, KASSERT fires. Without INVARIANTS the new sendfile won't panic, but would report incorrect amount of bytes sent. o With this change, the headers copyin is moved down into the cycle, after the sbspace() check. The uio size is trimmed by socket space there, which fixes the overcommit problem and its consequences. o The compatibility handling for FreeBSD 4 sendfile headers API is pushed up the stack to syscall wrappers. This required a copy and paste of the code, but in turn this allowed to remove extra stack carried parameter from fo_sendfile_t, and embrace entire compat code into #ifdef. If in future we got more fo_sendfile_t function, the copy and paste level would even reduce. Reviewed by: emax, gallatin, Maxim Dounin <mdounin mdounin.ru> Tested by: Vitalij Satanivskij <satan ukr.net> Sponsored by: Netflix
2016-03-29 19:57:11 +00:00
* At the beginning of the first loop check if any headers
* are specified and copy them into mbufs. Reduce space in
* the socket buffer by the size of the header mbuf chain.
* Clear hdr_uio here and hdrlen at the end of the first loop.
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
*/
The sendfile(2) allows to send extra data from userspace before the file data (headers). Historically the size of the headers was not checked against the socket buffer space. Application could easily overcommit the socket buffer space. With the new sendfile (r293439) the problem remained, but a KASSERT was inserted that checked that amount of data written to the socket matches its space. In case when size of headers is bigger that socket space, KASSERT fires. Without INVARIANTS the new sendfile won't panic, but would report incorrect amount of bytes sent. o With this change, the headers copyin is moved down into the cycle, after the sbspace() check. The uio size is trimmed by socket space there, which fixes the overcommit problem and its consequences. o The compatibility handling for FreeBSD 4 sendfile headers API is pushed up the stack to syscall wrappers. This required a copy and paste of the code, but in turn this allowed to remove extra stack carried parameter from fo_sendfile_t, and embrace entire compat code into #ifdef. If in future we got more fo_sendfile_t function, the copy and paste level would even reduce. Reviewed by: emax, gallatin, Maxim Dounin <mdounin mdounin.ru> Tested by: Vitalij Satanivskij <satan ukr.net> Sponsored by: Netflix
2016-03-29 19:57:11 +00:00
if (hdr_uio != NULL && hdr_uio->uio_resid > 0) {
hdr_uio->uio_td = td;
hdr_uio->uio_rw = UIO_WRITE;
Fix regression from r297400, which truncates headers in case of low socket buffer and put a small optimization for low socket buffer case: - Do not hack uio_resid, and let m_uiotombuf() properly take care of it. This fixes truncation of headers at low buffer. - If headers ate all the space, jump right to the end of the cycle, to avoid doing single page I/O and allocating zero length mbuf. - Clear hdr_uio only if space is positive, which indicates that all uio was copied in. Reviewed by: pluknet, jtl, emax, rrs, lstewart, emax, gallatin, scottl
2016-09-22 20:34:44 +00:00
mh = m_uiotombuf(hdr_uio, M_WAITOK, space, 0, 0);
The sendfile(2) allows to send extra data from userspace before the file data (headers). Historically the size of the headers was not checked against the socket buffer space. Application could easily overcommit the socket buffer space. With the new sendfile (r293439) the problem remained, but a KASSERT was inserted that checked that amount of data written to the socket matches its space. In case when size of headers is bigger that socket space, KASSERT fires. Without INVARIANTS the new sendfile won't panic, but would report incorrect amount of bytes sent. o With this change, the headers copyin is moved down into the cycle, after the sbspace() check. The uio size is trimmed by socket space there, which fixes the overcommit problem and its consequences. o The compatibility handling for FreeBSD 4 sendfile headers API is pushed up the stack to syscall wrappers. This required a copy and paste of the code, but in turn this allowed to remove extra stack carried parameter from fo_sendfile_t, and embrace entire compat code into #ifdef. If in future we got more fo_sendfile_t function, the copy and paste level would even reduce. Reviewed by: emax, gallatin, Maxim Dounin <mdounin mdounin.ru> Tested by: Vitalij Satanivskij <satan ukr.net> Sponsored by: Netflix
2016-03-29 19:57:11 +00:00
hdrlen = m_length(mh, &mhtail);
space -= hdrlen;
Fix regression from r297400, which truncates headers in case of low socket buffer and put a small optimization for low socket buffer case: - Do not hack uio_resid, and let m_uiotombuf() properly take care of it. This fixes truncation of headers at low buffer. - If headers ate all the space, jump right to the end of the cycle, to avoid doing single page I/O and allocating zero length mbuf. - Clear hdr_uio only if space is positive, which indicates that all uio was copied in. Reviewed by: pluknet, jtl, emax, rrs, lstewart, emax, gallatin, scottl
2016-09-22 20:34:44 +00:00
/*
* If header consumed all the socket buffer space,
* don't waste CPU cycles and jump to the end.
*/
if (space == 0) {
sfio = NULL;
nios = 0;
goto prepend_header;
}
The sendfile(2) allows to send extra data from userspace before the file data (headers). Historically the size of the headers was not checked against the socket buffer space. Application could easily overcommit the socket buffer space. With the new sendfile (r293439) the problem remained, but a KASSERT was inserted that checked that amount of data written to the socket matches its space. In case when size of headers is bigger that socket space, KASSERT fires. Without INVARIANTS the new sendfile won't panic, but would report incorrect amount of bytes sent. o With this change, the headers copyin is moved down into the cycle, after the sbspace() check. The uio size is trimmed by socket space there, which fixes the overcommit problem and its consequences. o The compatibility handling for FreeBSD 4 sendfile headers API is pushed up the stack to syscall wrappers. This required a copy and paste of the code, but in turn this allowed to remove extra stack carried parameter from fo_sendfile_t, and embrace entire compat code into #ifdef. If in future we got more fo_sendfile_t function, the copy and paste level would even reduce. Reviewed by: emax, gallatin, Maxim Dounin <mdounin mdounin.ru> Tested by: Vitalij Satanivskij <satan ukr.net> Sponsored by: Netflix
2016-03-29 19:57:11 +00:00
hdr_uio = NULL;
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
if (vp != NULL) {
error = vn_lock(vp, LK_SHARED);
if (error != 0)
goto done;
error = VOP_GETATTR(vp, &va, td->td_ucred);
if (error != 0 || off >= va.va_size) {
VOP_UNLOCK(vp, 0);
goto done;
}
if (va.va_size != obj_size) {
obj_size = va.va_size;
Make sendfile(2) more robust against file change. This fixes a possible crash when the file shrinks. This also fixes sendfile(2) not sending more data in a case when the file grows, and the request is open-ended or specifies a size that is greater than old file size. PR: 217789 Reviewed by: gallatin MFC after: 10 days
2017-03-24 16:01:19 +00:00
rem = nbytes ?
omin(nbytes + offset, obj_size) : obj_size;
rem -= off;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
}
}
if (space > rem)
space = rem;
Add support for using unmapped mbufs with sendfile(2). This can be enabled at runtime via the kern.ipc.mb_use_ext_pgs sysctl. It is disabled by default. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:49:35 +00:00
else if (space > PAGE_SIZE) {
/*
* Use page boundaries when possible for large
* requests.
*/
if (off & PAGE_MASK)
space -= (PAGE_SIZE - (off & PAGE_MASK));
space = trunc_page(space);
if (off & PAGE_MASK)
space += (PAGE_SIZE - (off & PAGE_MASK));
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
npages = howmany(space + (off & PAGE_MASK), PAGE_SIZE);
/*
* Calculate maximum allowed number of pages for readahead
Add flag SF_USER_READAHEAD to sendfile(2). When specified, the syscall won't do any speculations about readahead, and use exactly the amount of readahead specified by user. E.g. setting SF_FLAGS(0, SF_USER_READAHEAD) will guarantee that no readahead at all will be performed.
2016-11-17 21:36:18 +00:00
* at this iteration. If SF_USER_READAHEAD was set, we don't
* do any heuristics and use exactly the value supplied by
* application. Otherwise, we allow readahead up to "rem".
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
* If application wants more, let it be, but there is no
* reason to go above MAXPHYS. Also check against "obj_size",
* since vm_pager_has_page() can hint beyond EOF.
*/
Add flag SF_USER_READAHEAD to sendfile(2). When specified, the syscall won't do any speculations about readahead, and use exactly the amount of readahead specified by user. E.g. setting SF_FLAGS(0, SF_USER_READAHEAD) will guarantee that no readahead at all will be performed.
2016-11-17 21:36:18 +00:00
if (flags & SF_USER_READAHEAD) {
rhpages = SF_READAHEAD(flags);
} else {
rhpages = howmany(rem + (off & PAGE_MASK), PAGE_SIZE) -
npages;
rhpages += SF_READAHEAD(flags);
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
rhpages = min(howmany(MAXPHYS, PAGE_SIZE), rhpages);
rhpages = min(howmany(obj_size - trunc_page(off), PAGE_SIZE) -
npages, rhpages);
sfio = malloc(sizeof(struct sf_io) +
npages * sizeof(vm_page_t), M_TEMP, M_WAITOK);
refcount_init(&sfio->nios, 1);
Use soref() in sendfile(2) instead fhold() to reference a socket. The problem is that fdrop() requires syscall context, as it may enter sleep in some cases. The reason to use it in the original non-blocking sendfile implementation, was to avoid use of global ACCEPT_LOCK() on every I/O completion. Now in head sorele() no longer requires this lock.
2017-09-13 22:11:05 +00:00
sfio->so = so;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
sfio->error = 0;
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
error = sendfile_swapin(obj, sfio, &nios, off, space, npages,
rhpages, flags);
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error This is a partial merge of 350144 from projects/fuse2 PR: 236466 Reviewed by: markj MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21095
2019-07-29 20:50:26 +00:00
if (error != 0) {
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
if (vp != NULL)
VOP_UNLOCK(vp, 0);
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error This is a partial merge of 350144 from projects/fuse2 PR: 236466 Reviewed by: markj MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21095
2019-07-29 20:50:26 +00:00
free(sfio, M_TEMP);
sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error PR: 236466 Sponsored by: The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
goto done;
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
/*
* Loop and construct maximum sized mbuf chain to be bulk
* dumped into socket buffer.
*/
pa = sfio->pa;
Add support for using unmapped mbufs with sendfile(2). This can be enabled at runtime via the kern.ipc.mb_use_ext_pgs sysctl. It is disabled by default. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:49:35 +00:00
/*
* Use unmapped mbufs if enabled for TCP. Unmapped
* bufs are restricted to TCP as that is what has been
* tested. In particular, unmapped mbufs have not
* been tested with UNIX-domain sockets.
*/
if (mb_use_ext_pgs &&
so->so_proto->pr_protocol == IPPROTO_TCP) {
use_ext_pgs = true;
max_pgs = MBUF_PEXT_MAX_PGS;
/* Start at last index, to wrap on first use. */
ext_pgs_idx = max_pgs - 1;
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
for (int i = 0; i < npages; i++) {
struct mbuf *m0;
/*
* If a page wasn't grabbed successfully, then
* trim the array. Can happen only with SF_NODISKIO.
*/
if (pa[i] == NULL) {
SFSTAT_INC(sf_busy);
fixspace(npages, i, off, &space);
npages = i;
softerr = EBUSY;
break;
}
Add support for using unmapped mbufs with sendfile(2). This can be enabled at runtime via the kern.ipc.mb_use_ext_pgs sysctl. It is disabled by default. Submitted by: gallatin (earlier version) Reviewed by: gallatin, hselasky, rrs Relnotes: yes Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D20616
2019-06-29 00:49:35 +00:00
if (use_ext_pgs) {
off_t xfs;
ext_pgs_idx++;
if (ext_pgs_idx == max_pgs) {
m0 = mb_alloc_ext_pgs(M_WAITOK, false,
sendfile_free_mext_pg);
if (flags & SF_NOCACHE) {
m0->m_ext.ext_flags |=
EXT_FLAG_NOCACHE;
/*
* See comment below regarding
* ignoring SF_NOCACHE for the
* last page.
*/
if ((npages - i <= max_pgs) &&
((off + space) & PAGE_MASK) &&
(rem > space || rhpages > 0))
m0->m_ext.ext_flags |=
EXT_FLAG_CACHE_LAST;
}
if (sfs != NULL) {
m0->m_ext.ext_flags |=
EXT_FLAG_SYNC;
m0->m_ext.ext_arg2 = sfs;
mtx_lock(&sfs->mtx);
sfs->count++;
mtx_unlock(&sfs->mtx);
}
ext_pgs = m0->m_ext.ext_pgs;
if (i == 0)
sfio->m = m0;
ext_pgs_idx = 0;
/* Append to mbuf chain. */
if (mtail != NULL)
mtail->m_next = m0;
else
m = m0;
mtail = m0;
ext_pgs->first_pg_off =
vmoff(i, off) & PAGE_MASK;
}
if (nios) {
mtail->m_flags |= M_NOTREADY;
ext_pgs->nrdy++;
}
ext_pgs->pa[ext_pgs_idx] = VM_PAGE_TO_PHYS(pa[i]);
ext_pgs->npgs++;
xfs = xfsize(i, npages, off, space);
ext_pgs->last_pg_len = xfs;
MBUF_EXT_PGS_ASSERT_SANITY(ext_pgs);
mtail->m_len += xfs;
mtail->m_ext.ext_size += PAGE_SIZE;
continue;
}
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
/*
* Get a sendfile buf. When allocating the
* first buffer for mbuf chain, we usually
* wait as long as necessary, but this wait
* can be interrupted. For consequent
* buffers, do not sleep, since several
* threads might exhaust the buffers and then
* deadlock.
*/
sf = sf_buf_alloc(pa[i],
m != NULL ? SFB_NOWAIT : SFB_CATCH);
if (sf == NULL) {
SFSTAT_INC(sf_allocfail);
for (int j = i; j < npages; j++) {
vm_page_lock(pa[j]);
vm_page_unwire(pa[j], PQ_INACTIVE);
vm_page_unlock(pa[j]);
}
if (m == NULL)
softerr = ENOBUFS;
fixspace(npages, i, off, &space);
npages = i;
break;
}
m0 = m_get(M_WAITOK, MT_DATA);
m0->m_ext.ext_buf = (char *)sf_buf_kva(sf);
m0->m_ext.ext_size = PAGE_SIZE;
m0->m_ext.ext_arg1 = sf;
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
m0->m_ext.ext_type = EXT_SFBUF;
m0->m_ext.ext_flags = EXT_FLAG_EMBREF;
m0->m_ext.ext_free = sendfile_free_mext;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
/*
* SF_NOCACHE sets the page as being freed upon send.
* However, we ignore it for the last page in 'space',
* if the page is truncated, and we got more data to
* send (rem > space), or if we have readahead
* configured (rhpages > 0).
*/
Improvements to sendfile(2) mbuf free routine. o Fall back to default m_ext free mech, using function pointer in m_ext_free, and remove sf_ext_free() called directly from mbuf code. Testing on modern CPUs showed no regression. o Provide internally used flag EXT_FLAG_SYNC, to mark that I/O uses SF_SYNC flag. Lack of the flag allows us not to dereference ext_arg2, saving from a cache line miss. o Create function sendfile_free_page() that later will be used, for multi-page mbufs. For now compiler will inline it into sendfile_free_mext(). In collaboration with: gallatin Differential Revision: https://reviews.freebsd.org/D12615
2017-10-09 21:06:16 +00:00
if ((flags & SF_NOCACHE) &&
(i != npages - 1 ||
!((off + space) & PAGE_MASK) ||
!(rem > space || rhpages > 0)))
m0->m_ext.ext_flags |= EXT_FLAG_NOCACHE;
if (sfs != NULL) {
m0->m_ext.ext_flags |= EXT_FLAG_SYNC;
m0->m_ext.ext_arg2 = sfs;
mtx_lock(&sfs->mtx);
sfs->count++;
mtx_unlock(&sfs->mtx);
}
New way to manage reference counting of mbuf external storage. The m_ext.ext_cnt pointer becomes a union. It can now hold the refcount value itself. To tell that m_ext.ext_flags flag EXT_FLAG_EMBREF is used. The first mbuf to attach a cluster stores the refcount. The further mbufs to reference the cluster point at refcount in the first mbuf. The first mbuf is freed only when the last reference is freed. The benefit over refcounts stored in separate slabs is that now refcounts of different, unrelated mbufs do not share a cache line. For EXT_EXTREF mbufs the zone_ext_refcnt is no longer needed, and m_extadd() becomes void, making widely used M_EXTADD macro safe. For EXT_SFBUF mbufs the sf_ext_ref() is removed, which was an optimization exactly against the cache aliasing problem with regular refcounting. Discussed with: rrs, rwatson, gnn, hiren, sbruno, np Reviewed by: rrs Differential Revision: https://reviews.freebsd.org/D5396 Sponsored by: Netflix
2016-03-01 00:17:14 +00:00
m0->m_ext.ext_count = 1;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
m0->m_flags |= (M_EXT | M_RDONLY);
if (nios)
m0->m_flags |= M_NOTREADY;
m0->m_data = (char *)sf_buf_kva(sf) +
(vmoff(i, off) & PAGE_MASK);
m0->m_len = xfsize(i, npages, off, space);
if (i == 0)
sfio->m = m0;
/* Append to mbuf chain. */
if (mtail != NULL)
mtail->m_next = m0;
else
m = m0;
mtail = m0;
}
if (vp != NULL)
VOP_UNLOCK(vp, 0);
/* Keep track of bytes processed. */
off += space;
rem -= space;
/* Prepend header, if any. */
if (hdrlen) {
Fix regression from r297400, which truncates headers in case of low socket buffer and put a small optimization for low socket buffer case: - Do not hack uio_resid, and let m_uiotombuf() properly take care of it. This fixes truncation of headers at low buffer. - If headers ate all the space, jump right to the end of the cycle, to avoid doing single page I/O and allocating zero length mbuf. - Clear hdr_uio only if space is positive, which indicates that all uio was copied in. Reviewed by: pluknet, jtl, emax, rrs, lstewart, emax, gallatin, scottl
2016-09-22 20:34:44 +00:00
prepend_header:
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
mhtail->m_next = m;
m = mh;
mh = NULL;
}
if (m == NULL) {
KASSERT(softerr, ("%s: m NULL, no error", __func__));
error = softerr;
free(sfio, M_TEMP);
goto done;
}
/* Add the buffer chain to the socket buffer. */
KASSERT(m_length(m, NULL) == space + hdrlen,
("%s: mlen %u space %d hdrlen %d",
__func__, m_length(m, NULL), space, hdrlen));
CURVNET_SET(so->so_vnet);
if (nios == 0) {
/*
* If sendfile_swapin() didn't initiate any I/Os,
* which happens if all data is cached in VM, then
* we can send data right now without the
* PRUS_NOTREADY flag.
*/
free(sfio, M_TEMP);
error = (*so->so_proto->pr_usrreqs->pru_send)
(so, 0, m, NULL, NULL, td);
} else {
sfio->npages = npages;
Use soref() in sendfile(2) instead fhold() to reference a socket. The problem is that fdrop() requires syscall context, as it may enter sleep in some cases. The reason to use it in the original non-blocking sendfile implementation, was to avoid use of global ACCEPT_LOCK() on every I/O completion. Now in head sorele() no longer requires this lock.
2017-09-13 22:11:05 +00:00
soref(so);
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
error = (*so->so_proto->pr_usrreqs->pru_send)
(so, PRUS_NOTREADY, m, NULL, NULL, td);
sendfile_iodone(sfio, NULL, 0, 0);
}
CURVNET_RESTORE();
m = NULL; /* pru_send always consumes */
if (error)
goto done;
sbytes += space + hdrlen;
if (hdrlen)
hdrlen = 0;
if (softerr) {
error = softerr;
goto done;
}
}
/*
* Send trailers. Wimp out and use writev(2).
*/
if (trl_uio != NULL) {
sbunlock(&so->so_snd);
error = kern_writev(td, sockfd, trl_uio);
if (error == 0)
sbytes += td->td_retval[0];
goto out;
}
done:
sbunlock(&so->so_snd);
out:
/*
* If there was no error we have to clear td->td_retval[0]
* because it may have been set by writev.
*/
if (error == 0) {
td->td_retval[0] = 0;
}
if (sent != NULL) {
(*sent) = sbytes;
}
if (obj != NULL)
vm_object_deallocate(obj);
if (so)
fdrop(sock_fp, td);
if (m)
m_freem(m);
if (mh)
m_freem(mh);
if (sfs != NULL) {
mtx_lock(&sfs->mtx);
if (sfs->count != 0)
cv_wait(&sfs->cv, &sfs->mtx);
KASSERT(sfs->count == 0, ("sendfile sync still busy"));
cv_destroy(&sfs->cv);
mtx_destroy(&sfs->mtx);
free(sfs, M_TEMP);
}
if (error == ERESTART)
error = EINTR;
return (error);
}
static int
sendfile(struct thread *td, struct sendfile_args *uap, int compat)
{
struct sf_hdtr hdtr;
struct uio *hdr_uio, *trl_uio;
struct file *fp;
off_t sbytes;
int error;
/*
* File offset must be positive. If it goes beyond EOF
* we send only the header/trailer and no payload data.
*/
if (uap->offset < 0)
return (EINVAL);
Plug uninitialized stack variable leak in sendfile(2). Reported by: Ilja Van Sprundel <ivansprundel ioactive.com> Submitted by: Domagoj Stolfa <domagoj.stolfa gmail.com> MFC after: 1 week Security: uninitialized stack variable leak
2017-08-09 17:48:38 +00:00
sbytes = 0;
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
hdr_uio = trl_uio = NULL;
if (uap->hdtr != NULL) {
error = copyin(uap->hdtr, &hdtr, sizeof(hdtr));
if (error != 0)
goto out;
if (hdtr.headers != NULL) {
error = copyinuio(hdtr.headers, hdtr.hdr_cnt,
&hdr_uio);
if (error != 0)
goto out;
The sendfile(2) allows to send extra data from userspace before the file data (headers). Historically the size of the headers was not checked against the socket buffer space. Application could easily overcommit the socket buffer space. With the new sendfile (r293439) the problem remained, but a KASSERT was inserted that checked that amount of data written to the socket matches its space. In case when size of headers is bigger that socket space, KASSERT fires. Without INVARIANTS the new sendfile won't panic, but would report incorrect amount of bytes sent. o With this change, the headers copyin is moved down into the cycle, after the sbspace() check. The uio size is trimmed by socket space there, which fixes the overcommit problem and its consequences. o The compatibility handling for FreeBSD 4 sendfile headers API is pushed up the stack to syscall wrappers. This required a copy and paste of the code, but in turn this allowed to remove extra stack carried parameter from fo_sendfile_t, and embrace entire compat code into #ifdef. If in future we got more fo_sendfile_t function, the copy and paste level would even reduce. Reviewed by: emax, gallatin, Maxim Dounin <mdounin mdounin.ru> Tested by: Vitalij Satanivskij <satan ukr.net> Sponsored by: Netflix
2016-03-29 19:57:11 +00:00
#ifdef COMPAT_FREEBSD4
/*
* In FreeBSD < 5.0 the nbytes to send also included
* the header. If compat is specified subtract the
* header size from nbytes.
*/
if (compat) {
if (uap->nbytes > hdr_uio->uio_resid)
uap->nbytes -= hdr_uio->uio_resid;
else
uap->nbytes = 0;
}
#endif
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
}
if (hdtr.trailers != NULL) {
error = copyinuio(hdtr.trailers, hdtr.trl_cnt,
&trl_uio);
if (error != 0)
goto out;
}
}
AUDIT_ARG_FD(uap->fd);
/*
* sendfile(2) can start at any offset within a file so we require
* CAP_READ+CAP_SEEK = CAP_PREAD.
*/
Eliminate the overhead of gratuitous repeated reinitialization of cap_rights - Add macros to allow preinitialization of cap_rights_t. - Convert most commonly used code paths to use preinitialized cap_rights_t. A 3.6% speedup in fstat was measured with this change. Reported by: mjg Reviewed by: oshogbo Approved by: sbruno MFC after: 1 month
2018-05-09 18:47:24 +00:00
if ((error = fget_read(td, uap->fd, &cap_pread_rights, &fp)) != 0)
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
goto out;
error = fo_sendfile(fp, uap->s, hdr_uio, trl_uio, uap->offset,
The sendfile(2) allows to send extra data from userspace before the file data (headers). Historically the size of the headers was not checked against the socket buffer space. Application could easily overcommit the socket buffer space. With the new sendfile (r293439) the problem remained, but a KASSERT was inserted that checked that amount of data written to the socket matches its space. In case when size of headers is bigger that socket space, KASSERT fires. Without INVARIANTS the new sendfile won't panic, but would report incorrect amount of bytes sent. o With this change, the headers copyin is moved down into the cycle, after the sbspace() check. The uio size is trimmed by socket space there, which fixes the overcommit problem and its consequences. o The compatibility handling for FreeBSD 4 sendfile headers API is pushed up the stack to syscall wrappers. This required a copy and paste of the code, but in turn this allowed to remove extra stack carried parameter from fo_sendfile_t, and embrace entire compat code into #ifdef. If in future we got more fo_sendfile_t function, the copy and paste level would even reduce. Reviewed by: emax, gallatin, Maxim Dounin <mdounin mdounin.ru> Tested by: Vitalij Satanivskij <satan ukr.net> Sponsored by: Netflix
2016-03-29 19:57:11 +00:00
uap->nbytes, &sbytes, uap->flags, td);
- Separate sendfile(2) implementation from uipc_syscalls.c into separate file. Claim my copyright. - Provide more comments, better function and structure names. - Sort out unneeded includes from resulting two files. No functional changes.
2016-01-22 02:23:18 +00:00
fdrop(fp, td);
if (uap->sbytes != NULL)
copyout(&sbytes, uap->sbytes, sizeof(off_t));
out:
free(hdr_uio, M_IOV);
free(trl_uio, M_IOV);
return (error);
}
/*
* sendfile(2)
*
* int sendfile(int fd, int s, off_t offset, size_t nbytes,
* struct sf_hdtr *hdtr, off_t *sbytes, int flags)
*
* Send a file specified by 'fd' and starting at 'offset' to a socket
* specified by 's'. Send only 'nbytes' of the file or until EOF if nbytes ==
* 0. Optionally add a header and/or trailer to the socket output. If
* specified, write the total number of bytes sent into *sbytes.
*/
int
sys_sendfile(struct thread *td, struct sendfile_args *uap)
{
return (sendfile(td, uap, 0));
}
#ifdef COMPAT_FREEBSD4
int
freebsd4_sendfile(struct thread *td, struct freebsd4_sendfile_args *uap)
{
struct sendfile_args args;
args.fd = uap->fd;
args.s = uap->s;
args.offset = uap->offset;
args.nbytes = uap->nbytes;
args.hdtr = uap->hdtr;
args.sbytes = uap->sbytes;
args.flags = uap->flags;
return (sendfile(td, &args, 1));
}
#endif /* COMPAT_FREEBSD4 */
Reference in New Issue Copy Permalink