d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
a53a43d988
freebsd-dev/sys/kern/kern_physio.c

230 lines
6.2 KiB
C
Raw Normal View History

/* -> /*- for copyright notices, minor format tweaks as necessary
2005-01-06 23:35:40 +00:00
/*-
sys/kern: adoption of SPDX licensing ID tags. Mainly focus on files that use BSD 2-Clause license, however the tool I was using misidentified many licenses so this was mostly a manual - error prone - task. The Software Package Data Exchange (SPDX) group provides a specification to make it easier for automated tools to detect and summarize well known opensource licenses. We are gradually adopting the specification, noting that the tags are considered only advisory and do not, in any way, superceed or replace the license texts.
2017-11-27 15:20:12 +00:00
* SPDX-License-Identifier: BSD-4-Clause
*
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
* Copyright (c) 1994 John S. Dyson
* All rights reserved.
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
* notice immediately at the beginning of the file, without modification,
* this list of conditions, and the following disclaimer.
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
* 3. Absolutely no warranty of function or purpose is made by the author
* John S. Dyson.
* 4. Modifications may be freely made to this file if the above conditions
* are met.
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
*/
Use __FBSDID().
2003-06-11 00:56:59 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/param.h>
#include <sys/systm.h>
Separate the struct bio related stuff out of <sys/buf.h> into <sys/bio.h>. <sys/bio.h> is now a prerequisite for <sys/buf.h> but it shall not be made a nested include according to bdes teachings on the subject of nested includes. Diskdrivers and similar stuff below specfs::strategy() should no longer need to include <sys/buf.> unless they need caching of data. Still a few bogus uses of struct buf to track down. Repocopy by: peter
2000-05-05 09:59:14 +00:00
#include <sys/bio.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/buf.h>
#include <sys/conf.h>
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
#include <sys/malloc.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
#include <sys/proc.h>
Add four new RCTL resources - readbps, readiops, writebps and writeiops, for limiting disk (actually filesystem) IO. Note that in some cases these limits are not quite precise. It's ok, as long as it's within some reasonable bounds. Testing - and review of the code, in particular the VFS and VM parts - is very welcome. MFC after: 1 month Relnotes: yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D5080
2016-04-07 04:23:25 +00:00
#include <sys/racct.h>
Moved some #includes from <sys/param.h> nearer to where they are actually used.
1998-03-28 10:33:27 +00:00
#include <sys/uio.h>
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
#include <geom/geom.h>
Moved some #includes from <sys/param.h> nearer to where they are actually used.
1998-03-28 10:33:27 +00:00
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
#include <vm/vm.h>
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
#include <vm/vm_page.h>
Untangled the vm.h include file spaghetti.
1995-12-07 12:48:31 +00:00
#include <vm/vm_extern.h>
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
#include <vm/vm_map.h>
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
int
Do the dreaded s/dev_t/struct cdev */ Bump __FreeBSD_version accordingly.
2004-06-16 09:47:26 +00:00
physio(struct cdev *dev, struct uio *uio, int ioflag)
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
{
Add missing NULL check in physio(). When destroying a character device the si_devsw field is set to NULL before all references are gone, to indicate the character device is going away. This can cause a NULL-dereference fault inside physio(). The callers of physio() should own a thread reference on the cdev and if si_devsw is seen as non-NULL, it is usable during the execution of the function. Else an ENXIO error code is returned. Reviewed by: kib MFC after: 2 weeks
2015-10-29 13:53:37 +00:00
struct cdevsw *csw;
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
struct buf *pbuf;
struct bio *bp;
struct vm_page **pages;
Incorporated post 1.1.5 work from John Dyson. This includes performance improvements via the new routines pmap_qenter/pmap_qremove and pmap_kenter/ pmap_kremove. These routine allow fast mapping of pages for those architectures that have "normal" MMUs. Also included is a fix to the pageout daemon to properly check a queue end condition. Submitted by: John Dyson
1994-08-06 09:15:42 +00:00
caddr_t sa;
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
u_int iolen, poff;
int error, i, npages, maxpages;
vm_prot_t prot;
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
Add missing NULL check in physio(). When destroying a character device the si_devsw field is set to NULL before all references are gone, to indicate the character device is going away. This can cause a NULL-dereference fault inside physio(). The callers of physio() should own a thread reference on the cdev and if si_devsw is seen as non-NULL, it is usable during the execution of the function. Else an ENXIO error code is returned. Reviewed by: kib MFC after: 2 weeks
2015-10-29 13:53:37 +00:00
csw = dev->si_devsw;
physio: avoid uninitialized variables
2018-05-19 04:09:58 +00:00
npages = 0;
sa = NULL;
Add missing NULL check in physio(). When destroying a character device the si_devsw field is set to NULL before all references are gone, to indicate the character device is going away. This can cause a NULL-dereference fault inside physio(). The callers of physio() should own a thread reference on the cdev and if si_devsw is seen as non-NULL, it is usable during the execution of the function. Else an ENXIO error code is returned. Reviewed by: kib MFC after: 2 weeks
2015-10-29 13:53:37 +00:00
/* check if character device is being destroyed */
if (csw == NULL)
return (ENXIO);
Give physio a makeover. - Let physio take read/write compatible args and have it use uio->uio_rw to determine the direction. - physread/physwrite are now #defines for physio - Remove the inversly named minphys(), dev->si_iosize_max takes over. - Physio() always uses pbufs. - Fix the check for non page-aligned transfers, now only unaligned transfers larger than (MAXPHYS - PAGE_SIZE) get fragmented (only interesting for tapes using max blocksize). - General wash-and-clean of code. Constructive input from: bde
1999-10-09 19:44:32 +00:00
/* XXX: sanity check */
if(dev->si_iosize_max < PAGE_SIZE) {
printf("WARNING: %s si_iosize_max=%d, using DFLTPHYS.\n",
devtoname(dev), dev->si_iosize_max);
dev->si_iosize_max = DFLTPHYS;
}
Add support to physio(9) for devices that don't want I/O split and configure sa(4) to request no I/O splitting by default. For tape devices, the user needs to be able to clearly understand what blocksize is actually being used when writing to a tape device. The previous behavior of physio(9) was that it would split up any I/O that was too large for the device, or too large to fit into MAXPHYS. This means that if, for instance, the user wrote a 1MB block to a tape device, and MAXPHYS was 128KB, the 1MB write would be split into 8 128K chunks. This would be done without informing the user. This has suboptimal effects, especially when trying to communicate status to the user. In the event of an error writing to a tape (e.g. physical end of tape) in the middle of a 1MB block that has been split into 8 pieces, the user could have the first two 128K pieces written successfully, the third returned with an error, and the last 5 returned with 0 bytes written. If the user is using a standard write(2) system call, all he will see is the ENOSPC error. He won't have a clue how much actually got written. (With a writev(2) system call, he should be able to determine how much got written in addition to the error.) The solution is to prevent physio(9) from splitting the I/O. The new cdev flag, SI_NOSPLIT, tells physio that the driver does not want I/O to be split beforehand. Although the sa(4) driver now enables SI_NOSPLIT by default, that can be disabled by two loader tunables for now. It will not be configurable starting in FreeBSD 11.0. kern.cam.sa.allow_io_split allows the user to configure I/O splitting for all sa(4) driver instances. kern.cam.sa.%d.allow_io_split allows the user to configure I/O splitting for a specific sa(4) instance. There are also now three sa(4) driver sysctl variables that let the users see some sa(4) driver values. kern.cam.sa.%d.allow_io_split shows whether I/O splitting is turned on. kern.cam.sa.%d.maxio shows the maximum I/O size allowed by kernel configuration parameters (e.g. MAXPHYS, DFLTPHYS) and the capabilities of the controller. kern.cam.sa.%d.cpi_maxio shows the maximum I/O size supported by the controller. Note that a better long term solution would be to implement support for chaining buffers, so that that MAXPHYS is no longer a limiting factor for I/O size to tape and disk devices. At that point, the controller and the tape drive would become the limiting factors. sys/conf.h: Add a new cdev flag, SI_NOSPLIT, that allows a driver to tell physio not to split up I/O. sys/param.h: Bump __FreeBSD_version to 1000049 for the addition of the SI_NOSPLIT cdev flag. kern_physio.c: If the SI_NOSPLIT flag is set on the cdev, return any I/O that is larger than si_iosize_max or MAXPHYS, has more than one segment, or would have to be split because of misalignment with EFBIG. (File too large). In the event of an error, print a console message to give the user a clue about what happened. scsi_sa.c: Set the SI_NOSPLIT cdev flag on the devices created for the sa(4) driver by default. Add tunables to control whether we allow I/O splitting in physio(9). Explain in the comments that allowing I/O splitting will be deprecated for the sa(4) driver in FreeBSD 11.0. Add sysctl variables to display the maximum I/O size we can do (which could be further limited by read block limits) and the maximum I/O size that the controller can do. Limit our maximum I/O size (recorded in the cdev's si_iosize_max) by MAXPHYS. This isn't strictly necessary, because physio(9) will limit it to MAXPHYS, but it will provide some clarity for the application. Record the controller's maximum I/O size reported in the Path Inquiry CCB. sa.4: Document the block size behavior, and explain that the option of allowing physio(9) to split the I/O will disappear in FreeBSD 11.0. Sponsored by: Spectra Logic
2013-08-24 04:52:22 +00:00
/*
* If the driver does not want I/O to be split, that means that we
* need to reject any requests that will not fit into one buffer.
*/
Fix some issues in change 254760 pointed out by Bruce Evans: - Remove excessive parenthesis - Use KNF continuation indentation - Cut down on excessive continuation lines - More consistent style in messages - Use uprintf() instead of printf() Submitted by: bde
2013-08-29 16:41:40 +00:00
if (dev->si_flags & SI_NOSPLIT &&
(uio->uio_resid > dev->si_iosize_max || uio->uio_resid > MAXPHYS ||
uio->uio_iovcnt > 1)) {
Add support to physio(9) for devices that don't want I/O split and configure sa(4) to request no I/O splitting by default. For tape devices, the user needs to be able to clearly understand what blocksize is actually being used when writing to a tape device. The previous behavior of physio(9) was that it would split up any I/O that was too large for the device, or too large to fit into MAXPHYS. This means that if, for instance, the user wrote a 1MB block to a tape device, and MAXPHYS was 128KB, the 1MB write would be split into 8 128K chunks. This would be done without informing the user. This has suboptimal effects, especially when trying to communicate status to the user. In the event of an error writing to a tape (e.g. physical end of tape) in the middle of a 1MB block that has been split into 8 pieces, the user could have the first two 128K pieces written successfully, the third returned with an error, and the last 5 returned with 0 bytes written. If the user is using a standard write(2) system call, all he will see is the ENOSPC error. He won't have a clue how much actually got written. (With a writev(2) system call, he should be able to determine how much got written in addition to the error.) The solution is to prevent physio(9) from splitting the I/O. The new cdev flag, SI_NOSPLIT, tells physio that the driver does not want I/O to be split beforehand. Although the sa(4) driver now enables SI_NOSPLIT by default, that can be disabled by two loader tunables for now. It will not be configurable starting in FreeBSD 11.0. kern.cam.sa.allow_io_split allows the user to configure I/O splitting for all sa(4) driver instances. kern.cam.sa.%d.allow_io_split allows the user to configure I/O splitting for a specific sa(4) instance. There are also now three sa(4) driver sysctl variables that let the users see some sa(4) driver values. kern.cam.sa.%d.allow_io_split shows whether I/O splitting is turned on. kern.cam.sa.%d.maxio shows the maximum I/O size allowed by kernel configuration parameters (e.g. MAXPHYS, DFLTPHYS) and the capabilities of the controller. kern.cam.sa.%d.cpi_maxio shows the maximum I/O size supported by the controller. Note that a better long term solution would be to implement support for chaining buffers, so that that MAXPHYS is no longer a limiting factor for I/O size to tape and disk devices. At that point, the controller and the tape drive would become the limiting factors. sys/conf.h: Add a new cdev flag, SI_NOSPLIT, that allows a driver to tell physio not to split up I/O. sys/param.h: Bump __FreeBSD_version to 1000049 for the addition of the SI_NOSPLIT cdev flag. kern_physio.c: If the SI_NOSPLIT flag is set on the cdev, return any I/O that is larger than si_iosize_max or MAXPHYS, has more than one segment, or would have to be split because of misalignment with EFBIG. (File too large). In the event of an error, print a console message to give the user a clue about what happened. scsi_sa.c: Set the SI_NOSPLIT cdev flag on the devices created for the sa(4) driver by default. Add tunables to control whether we allow I/O splitting in physio(9). Explain in the comments that allowing I/O splitting will be deprecated for the sa(4) driver in FreeBSD 11.0. Add sysctl variables to display the maximum I/O size we can do (which could be further limited by read block limits) and the maximum I/O size that the controller can do. Limit our maximum I/O size (recorded in the cdev's si_iosize_max) by MAXPHYS. This isn't strictly necessary, because physio(9) will limit it to MAXPHYS, but it will provide some clarity for the application. Record the controller's maximum I/O size reported in the Path Inquiry CCB. sa.4: Document the block size behavior, and explain that the option of allowing physio(9) to split the I/O will disappear in FreeBSD 11.0. Sponsored by: Spectra Logic
2013-08-24 04:52:22 +00:00
/*
* Tell the user why his I/O was rejected.
*/
if (uio->uio_resid > dev->si_iosize_max)
Fix some issues in change 254760 pointed out by Bruce Evans: - Remove excessive parenthesis - Use KNF continuation indentation - Cut down on excessive continuation lines - More consistent style in messages - Use uprintf() instead of printf() Submitted by: bde
2013-08-29 16:41:40 +00:00
uprintf("%s: request size=%zd > si_iosize_max=%d; "
Add support to physio(9) for devices that don't want I/O split and configure sa(4) to request no I/O splitting by default. For tape devices, the user needs to be able to clearly understand what blocksize is actually being used when writing to a tape device. The previous behavior of physio(9) was that it would split up any I/O that was too large for the device, or too large to fit into MAXPHYS. This means that if, for instance, the user wrote a 1MB block to a tape device, and MAXPHYS was 128KB, the 1MB write would be split into 8 128K chunks. This would be done without informing the user. This has suboptimal effects, especially when trying to communicate status to the user. In the event of an error writing to a tape (e.g. physical end of tape) in the middle of a 1MB block that has been split into 8 pieces, the user could have the first two 128K pieces written successfully, the third returned with an error, and the last 5 returned with 0 bytes written. If the user is using a standard write(2) system call, all he will see is the ENOSPC error. He won't have a clue how much actually got written. (With a writev(2) system call, he should be able to determine how much got written in addition to the error.) The solution is to prevent physio(9) from splitting the I/O. The new cdev flag, SI_NOSPLIT, tells physio that the driver does not want I/O to be split beforehand. Although the sa(4) driver now enables SI_NOSPLIT by default, that can be disabled by two loader tunables for now. It will not be configurable starting in FreeBSD 11.0. kern.cam.sa.allow_io_split allows the user to configure I/O splitting for all sa(4) driver instances. kern.cam.sa.%d.allow_io_split allows the user to configure I/O splitting for a specific sa(4) instance. There are also now three sa(4) driver sysctl variables that let the users see some sa(4) driver values. kern.cam.sa.%d.allow_io_split shows whether I/O splitting is turned on. kern.cam.sa.%d.maxio shows the maximum I/O size allowed by kernel configuration parameters (e.g. MAXPHYS, DFLTPHYS) and the capabilities of the controller. kern.cam.sa.%d.cpi_maxio shows the maximum I/O size supported by the controller. Note that a better long term solution would be to implement support for chaining buffers, so that that MAXPHYS is no longer a limiting factor for I/O size to tape and disk devices. At that point, the controller and the tape drive would become the limiting factors. sys/conf.h: Add a new cdev flag, SI_NOSPLIT, that allows a driver to tell physio not to split up I/O. sys/param.h: Bump __FreeBSD_version to 1000049 for the addition of the SI_NOSPLIT cdev flag. kern_physio.c: If the SI_NOSPLIT flag is set on the cdev, return any I/O that is larger than si_iosize_max or MAXPHYS, has more than one segment, or would have to be split because of misalignment with EFBIG. (File too large). In the event of an error, print a console message to give the user a clue about what happened. scsi_sa.c: Set the SI_NOSPLIT cdev flag on the devices created for the sa(4) driver by default. Add tunables to control whether we allow I/O splitting in physio(9). Explain in the comments that allowing I/O splitting will be deprecated for the sa(4) driver in FreeBSD 11.0. Add sysctl variables to display the maximum I/O size we can do (which could be further limited by read block limits) and the maximum I/O size that the controller can do. Limit our maximum I/O size (recorded in the cdev's si_iosize_max) by MAXPHYS. This isn't strictly necessary, because physio(9) will limit it to MAXPHYS, but it will provide some clarity for the application. Record the controller's maximum I/O size reported in the Path Inquiry CCB. sa.4: Document the block size behavior, and explain that the option of allowing physio(9) to split the I/O will disappear in FreeBSD 11.0. Sponsored by: Spectra Logic
2013-08-24 04:52:22 +00:00
"cannot split request\n", devtoname(dev),
uio->uio_resid, dev->si_iosize_max);
if (uio->uio_resid > MAXPHYS)
Fix some issues in change 254760 pointed out by Bruce Evans: - Remove excessive parenthesis - Use KNF continuation indentation - Cut down on excessive continuation lines - More consistent style in messages - Use uprintf() instead of printf() Submitted by: bde
2013-08-29 16:41:40 +00:00
uprintf("%s: request size=%zd > MAXPHYS=%d; "
Add support to physio(9) for devices that don't want I/O split and configure sa(4) to request no I/O splitting by default. For tape devices, the user needs to be able to clearly understand what blocksize is actually being used when writing to a tape device. The previous behavior of physio(9) was that it would split up any I/O that was too large for the device, or too large to fit into MAXPHYS. This means that if, for instance, the user wrote a 1MB block to a tape device, and MAXPHYS was 128KB, the 1MB write would be split into 8 128K chunks. This would be done without informing the user. This has suboptimal effects, especially when trying to communicate status to the user. In the event of an error writing to a tape (e.g. physical end of tape) in the middle of a 1MB block that has been split into 8 pieces, the user could have the first two 128K pieces written successfully, the third returned with an error, and the last 5 returned with 0 bytes written. If the user is using a standard write(2) system call, all he will see is the ENOSPC error. He won't have a clue how much actually got written. (With a writev(2) system call, he should be able to determine how much got written in addition to the error.) The solution is to prevent physio(9) from splitting the I/O. The new cdev flag, SI_NOSPLIT, tells physio that the driver does not want I/O to be split beforehand. Although the sa(4) driver now enables SI_NOSPLIT by default, that can be disabled by two loader tunables for now. It will not be configurable starting in FreeBSD 11.0. kern.cam.sa.allow_io_split allows the user to configure I/O splitting for all sa(4) driver instances. kern.cam.sa.%d.allow_io_split allows the user to configure I/O splitting for a specific sa(4) instance. There are also now three sa(4) driver sysctl variables that let the users see some sa(4) driver values. kern.cam.sa.%d.allow_io_split shows whether I/O splitting is turned on. kern.cam.sa.%d.maxio shows the maximum I/O size allowed by kernel configuration parameters (e.g. MAXPHYS, DFLTPHYS) and the capabilities of the controller. kern.cam.sa.%d.cpi_maxio shows the maximum I/O size supported by the controller. Note that a better long term solution would be to implement support for chaining buffers, so that that MAXPHYS is no longer a limiting factor for I/O size to tape and disk devices. At that point, the controller and the tape drive would become the limiting factors. sys/conf.h: Add a new cdev flag, SI_NOSPLIT, that allows a driver to tell physio not to split up I/O. sys/param.h: Bump __FreeBSD_version to 1000049 for the addition of the SI_NOSPLIT cdev flag. kern_physio.c: If the SI_NOSPLIT flag is set on the cdev, return any I/O that is larger than si_iosize_max or MAXPHYS, has more than one segment, or would have to be split because of misalignment with EFBIG. (File too large). In the event of an error, print a console message to give the user a clue about what happened. scsi_sa.c: Set the SI_NOSPLIT cdev flag on the devices created for the sa(4) driver by default. Add tunables to control whether we allow I/O splitting in physio(9). Explain in the comments that allowing I/O splitting will be deprecated for the sa(4) driver in FreeBSD 11.0. Add sysctl variables to display the maximum I/O size we can do (which could be further limited by read block limits) and the maximum I/O size that the controller can do. Limit our maximum I/O size (recorded in the cdev's si_iosize_max) by MAXPHYS. This isn't strictly necessary, because physio(9) will limit it to MAXPHYS, but it will provide some clarity for the application. Record the controller's maximum I/O size reported in the Path Inquiry CCB. sa.4: Document the block size behavior, and explain that the option of allowing physio(9) to split the I/O will disappear in FreeBSD 11.0. Sponsored by: Spectra Logic
2013-08-24 04:52:22 +00:00
"cannot split request\n", devtoname(dev),
uio->uio_resid, MAXPHYS);
if (uio->uio_iovcnt > 1)
Fix some issues in change 254760 pointed out by Bruce Evans: - Remove excessive parenthesis - Use KNF continuation indentation - Cut down on excessive continuation lines - More consistent style in messages - Use uprintf() instead of printf() Submitted by: bde
2013-08-29 16:41:40 +00:00
uprintf("%s: request vectors=%d > 1; "
Add support to physio(9) for devices that don't want I/O split and configure sa(4) to request no I/O splitting by default. For tape devices, the user needs to be able to clearly understand what blocksize is actually being used when writing to a tape device. The previous behavior of physio(9) was that it would split up any I/O that was too large for the device, or too large to fit into MAXPHYS. This means that if, for instance, the user wrote a 1MB block to a tape device, and MAXPHYS was 128KB, the 1MB write would be split into 8 128K chunks. This would be done without informing the user. This has suboptimal effects, especially when trying to communicate status to the user. In the event of an error writing to a tape (e.g. physical end of tape) in the middle of a 1MB block that has been split into 8 pieces, the user could have the first two 128K pieces written successfully, the third returned with an error, and the last 5 returned with 0 bytes written. If the user is using a standard write(2) system call, all he will see is the ENOSPC error. He won't have a clue how much actually got written. (With a writev(2) system call, he should be able to determine how much got written in addition to the error.) The solution is to prevent physio(9) from splitting the I/O. The new cdev flag, SI_NOSPLIT, tells physio that the driver does not want I/O to be split beforehand. Although the sa(4) driver now enables SI_NOSPLIT by default, that can be disabled by two loader tunables for now. It will not be configurable starting in FreeBSD 11.0. kern.cam.sa.allow_io_split allows the user to configure I/O splitting for all sa(4) driver instances. kern.cam.sa.%d.allow_io_split allows the user to configure I/O splitting for a specific sa(4) instance. There are also now three sa(4) driver sysctl variables that let the users see some sa(4) driver values. kern.cam.sa.%d.allow_io_split shows whether I/O splitting is turned on. kern.cam.sa.%d.maxio shows the maximum I/O size allowed by kernel configuration parameters (e.g. MAXPHYS, DFLTPHYS) and the capabilities of the controller. kern.cam.sa.%d.cpi_maxio shows the maximum I/O size supported by the controller. Note that a better long term solution would be to implement support for chaining buffers, so that that MAXPHYS is no longer a limiting factor for I/O size to tape and disk devices. At that point, the controller and the tape drive would become the limiting factors. sys/conf.h: Add a new cdev flag, SI_NOSPLIT, that allows a driver to tell physio not to split up I/O. sys/param.h: Bump __FreeBSD_version to 1000049 for the addition of the SI_NOSPLIT cdev flag. kern_physio.c: If the SI_NOSPLIT flag is set on the cdev, return any I/O that is larger than si_iosize_max or MAXPHYS, has more than one segment, or would have to be split because of misalignment with EFBIG. (File too large). In the event of an error, print a console message to give the user a clue about what happened. scsi_sa.c: Set the SI_NOSPLIT cdev flag on the devices created for the sa(4) driver by default. Add tunables to control whether we allow I/O splitting in physio(9). Explain in the comments that allowing I/O splitting will be deprecated for the sa(4) driver in FreeBSD 11.0. Add sysctl variables to display the maximum I/O size we can do (which could be further limited by read block limits) and the maximum I/O size that the controller can do. Limit our maximum I/O size (recorded in the cdev's si_iosize_max) by MAXPHYS. This isn't strictly necessary, because physio(9) will limit it to MAXPHYS, but it will provide some clarity for the application. Record the controller's maximum I/O size reported in the Path Inquiry CCB. sa.4: Document the block size behavior, and explain that the option of allowing physio(9) to split the I/O will disappear in FreeBSD 11.0. Sponsored by: Spectra Logic
2013-08-24 04:52:22 +00:00
"cannot split request\n", devtoname(dev),
uio->uio_iovcnt);
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
return (EFBIG);
Add support to physio(9) for devices that don't want I/O split and configure sa(4) to request no I/O splitting by default. For tape devices, the user needs to be able to clearly understand what blocksize is actually being used when writing to a tape device. The previous behavior of physio(9) was that it would split up any I/O that was too large for the device, or too large to fit into MAXPHYS. This means that if, for instance, the user wrote a 1MB block to a tape device, and MAXPHYS was 128KB, the 1MB write would be split into 8 128K chunks. This would be done without informing the user. This has suboptimal effects, especially when trying to communicate status to the user. In the event of an error writing to a tape (e.g. physical end of tape) in the middle of a 1MB block that has been split into 8 pieces, the user could have the first two 128K pieces written successfully, the third returned with an error, and the last 5 returned with 0 bytes written. If the user is using a standard write(2) system call, all he will see is the ENOSPC error. He won't have a clue how much actually got written. (With a writev(2) system call, he should be able to determine how much got written in addition to the error.) The solution is to prevent physio(9) from splitting the I/O. The new cdev flag, SI_NOSPLIT, tells physio that the driver does not want I/O to be split beforehand. Although the sa(4) driver now enables SI_NOSPLIT by default, that can be disabled by two loader tunables for now. It will not be configurable starting in FreeBSD 11.0. kern.cam.sa.allow_io_split allows the user to configure I/O splitting for all sa(4) driver instances. kern.cam.sa.%d.allow_io_split allows the user to configure I/O splitting for a specific sa(4) instance. There are also now three sa(4) driver sysctl variables that let the users see some sa(4) driver values. kern.cam.sa.%d.allow_io_split shows whether I/O splitting is turned on. kern.cam.sa.%d.maxio shows the maximum I/O size allowed by kernel configuration parameters (e.g. MAXPHYS, DFLTPHYS) and the capabilities of the controller. kern.cam.sa.%d.cpi_maxio shows the maximum I/O size supported by the controller. Note that a better long term solution would be to implement support for chaining buffers, so that that MAXPHYS is no longer a limiting factor for I/O size to tape and disk devices. At that point, the controller and the tape drive would become the limiting factors. sys/conf.h: Add a new cdev flag, SI_NOSPLIT, that allows a driver to tell physio not to split up I/O. sys/param.h: Bump __FreeBSD_version to 1000049 for the addition of the SI_NOSPLIT cdev flag. kern_physio.c: If the SI_NOSPLIT flag is set on the cdev, return any I/O that is larger than si_iosize_max or MAXPHYS, has more than one segment, or would have to be split because of misalignment with EFBIG. (File too large). In the event of an error, print a console message to give the user a clue about what happened. scsi_sa.c: Set the SI_NOSPLIT cdev flag on the devices created for the sa(4) driver by default. Add tunables to control whether we allow I/O splitting in physio(9). Explain in the comments that allowing I/O splitting will be deprecated for the sa(4) driver in FreeBSD 11.0. Add sysctl variables to display the maximum I/O size we can do (which could be further limited by read block limits) and the maximum I/O size that the controller can do. Limit our maximum I/O size (recorded in the cdev's si_iosize_max) by MAXPHYS. This isn't strictly necessary, because physio(9) will limit it to MAXPHYS, but it will provide some clarity for the application. Record the controller's maximum I/O size reported in the Path Inquiry CCB. sa.4: Document the block size behavior, and explain that the option of allowing physio(9) to split the I/O will disappear in FreeBSD 11.0. Sponsored by: Spectra Logic
2013-08-24 04:52:22 +00:00
}
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
/*
* Keep the process UPAGES from being swapped. Processes swapped
* out while holding pbufs, used by swapper, may lead to deadlock.
*/
PHOLD(curproc);
bp = g_alloc_bio();
if (uio->uio_segflg != UIO_USERSPACE) {
pbuf = NULL;
pages = NULL;
} else if ((dev->si_flags & SI_UNMAPPED) && unmapped_buf_allowed) {
pbuf = NULL;
maxpages = btoc(MIN(uio->uio_resid, MAXPHYS)) + 1;
pages = malloc(sizeof(*pages) * maxpages, M_DEVBUF, M_WAITOK);
} else {
Allocate pager bufs from UMA instead of 80-ish mutex protected linked list. o In vm_pager_bufferinit() create pbuf_zone and start accounting on how many pbufs are we going to have set. In various subsystems that are going to utilize pbufs create private zones via call to pbuf_zsecond_create(). The latter calls uma_zsecond_create(), and sets a limit on created zone. After startup preallocate pbufs according to requirements of all pbuf zones. Subsystems that used to have a private limit with old allocator now have private pbuf zones: md(4), fusefs, NFS client, smbfs, VFS cluster, FFS, swap, vnode pager. The following subsystems use shared pbuf zone: cam(4), nvme(4), physio(9), aio(4). They should have their private limits, but changing that is out of scope of this commit. o Fetch tunable value of kern.nswbuf from init_param2() and while here move NSWBUF_MIN to opt_param.h and eliminate opt_swap.h, that was holding only this option. Default values aren't touched by this commit, but they probably should be reviewed wrt to modern hardware. This change removes a tight bottleneck from sendfile(2) operation, that uses pbufs in vnode pager. Other pagers also would benefit from faster allocation. Together with: gallatin Tested by: pho
2019-01-15 01:02:16 +00:00
pbuf = uma_zalloc(pbuf_zone, M_WAITOK);
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
sa = pbuf->b_data;
maxpages = btoc(MAXPHYS);
pages = pbuf->b_pages;
}
prot = VM_PROT_READ;
if (uio->uio_rw == UIO_READ)
prot |= VM_PROT_WRITE; /* Less backwards than it looks */
error = 0;
Convert buffer locking from using the B_BUSY and B_WANTED flags to using lockmgr locks. This commit should be functionally equivalent to the old semantics. That is, all buffer locking is done with LK_EXCLUSIVE requests. Changes to take advantage of LK_SHARED and LK_RECURSIVE will be done in future commits.
1999-06-26 02:47:16 +00:00
for (i = 0; i < uio->uio_iovcnt; i++) {
Add four new RCTL resources - readbps, readiops, writebps and writeiops, for limiting disk (actually filesystem) IO. Note that in some cases these limits are not quite precise. It's ok, as long as it's within some reasonable bounds. Testing - and review of the code, in particular the VFS and VM parts - is very welcome. MFC after: 1 month Relnotes: yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D5080
2016-04-07 04:23:25 +00:00
#ifdef RACCT
if (racct_enable) {
PROC_LOCK(curproc);
if (uio->uio_rw == UIO_READ) {
racct_add_force(curproc, RACCT_READBPS,
uio->uio_iov[i].iov_len);
racct_add_force(curproc, RACCT_READIOPS, 1);
} else {
racct_add_force(curproc, RACCT_WRITEBPS,
uio->uio_iov[i].iov_len);
racct_add_force(curproc, RACCT_WRITEIOPS, 1);
}
PROC_UNLOCK(curproc);
}
#endif /* RACCT */
Convert buffer locking from using the B_BUSY and B_WANTED flags to using lockmgr locks. This commit should be functionally equivalent to the old semantics. That is, all buffer locking is done with LK_EXCLUSIVE requests. Changes to take advantage of LK_SHARED and LK_RECURSIVE will be done in future commits.
1999-06-26 02:47:16 +00:00
while (uio->uio_iov[i].iov_len) {
Create an API to reset a struct bio (g_reset_bio). This is mandatory for all struct bio you get back from g_{new,alloc}_bio. Temporary bios that you create on the stack or elsewhere should use this before first use of the bio, and between uses of the bio. At the moment, it is nothing more than a wrapper around bzero, but that may change in the future. The wrapper also removes one place where we encode the size of struct bio in the KBI.
2016-02-17 17:16:02 +00:00
g_reset_bio(bp);
Account i/o done on cdevs. Reported and tested by: Adam Vande More <amvandemore gmail com> MFC after: 1 week
2010-11-25 20:05:11 +00:00
if (uio->uio_rw == UIO_READ) {
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
bp->bio_cmd = BIO_READ;
Account i/o done on cdevs. Reported and tested by: Adam Vande More <amvandemore gmail com> MFC after: 1 week
2010-11-25 20:05:11 +00:00
curthread->td_ru.ru_inblock++;
} else {
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
bp->bio_cmd = BIO_WRITE;
Account i/o done on cdevs. Reported and tested by: Adam Vande More <amvandemore gmail com> MFC after: 1 week
2010-11-25 20:05:11 +00:00
curthread->td_ru.ru_oublock++;
}
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
bp->bio_offset = uio->uio_offset;
bp->bio_data = uio->uio_iov[i].iov_base;
bp->bio_length = uio->uio_iov[i].iov_len;
if (bp->bio_length > dev->si_iosize_max)
bp->bio_length = dev->si_iosize_max;
if (bp->bio_length > MAXPHYS)
bp->bio_length = MAXPHYS;
/*
* Make sure the pbuf can map the request.
* The pbuf has kvasize = MAXPHYS, so a request
* larger than MAXPHYS - PAGE_SIZE must be
* page aligned or it will be fragmented.
Give physio a makeover. - Let physio take read/write compatible args and have it use uio->uio_rw to determine the direction. - physread/physwrite are now #defines for physio - Remove the inversly named minphys(), dev->si_iosize_max takes over. - Physio() always uses pbufs. - Fix the check for non page-aligned transfers, now only unaligned transfers larger than (MAXPHYS - PAGE_SIZE) get fragmented (only interesting for tapes using max blocksize). - General wash-and-clean of code. Constructive input from: bde
1999-10-09 19:44:32 +00:00
*/
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
poff = (vm_offset_t)bp->bio_data & PAGE_MASK;
if (pbuf && bp->bio_length + poff > pbuf->b_kvasize) {
Add support to physio(9) for devices that don't want I/O split and configure sa(4) to request no I/O splitting by default. For tape devices, the user needs to be able to clearly understand what blocksize is actually being used when writing to a tape device. The previous behavior of physio(9) was that it would split up any I/O that was too large for the device, or too large to fit into MAXPHYS. This means that if, for instance, the user wrote a 1MB block to a tape device, and MAXPHYS was 128KB, the 1MB write would be split into 8 128K chunks. This would be done without informing the user. This has suboptimal effects, especially when trying to communicate status to the user. In the event of an error writing to a tape (e.g. physical end of tape) in the middle of a 1MB block that has been split into 8 pieces, the user could have the first two 128K pieces written successfully, the third returned with an error, and the last 5 returned with 0 bytes written. If the user is using a standard write(2) system call, all he will see is the ENOSPC error. He won't have a clue how much actually got written. (With a writev(2) system call, he should be able to determine how much got written in addition to the error.) The solution is to prevent physio(9) from splitting the I/O. The new cdev flag, SI_NOSPLIT, tells physio that the driver does not want I/O to be split beforehand. Although the sa(4) driver now enables SI_NOSPLIT by default, that can be disabled by two loader tunables for now. It will not be configurable starting in FreeBSD 11.0. kern.cam.sa.allow_io_split allows the user to configure I/O splitting for all sa(4) driver instances. kern.cam.sa.%d.allow_io_split allows the user to configure I/O splitting for a specific sa(4) instance. There are also now three sa(4) driver sysctl variables that let the users see some sa(4) driver values. kern.cam.sa.%d.allow_io_split shows whether I/O splitting is turned on. kern.cam.sa.%d.maxio shows the maximum I/O size allowed by kernel configuration parameters (e.g. MAXPHYS, DFLTPHYS) and the capabilities of the controller. kern.cam.sa.%d.cpi_maxio shows the maximum I/O size supported by the controller. Note that a better long term solution would be to implement support for chaining buffers, so that that MAXPHYS is no longer a limiting factor for I/O size to tape and disk devices. At that point, the controller and the tape drive would become the limiting factors. sys/conf.h: Add a new cdev flag, SI_NOSPLIT, that allows a driver to tell physio not to split up I/O. sys/param.h: Bump __FreeBSD_version to 1000049 for the addition of the SI_NOSPLIT cdev flag. kern_physio.c: If the SI_NOSPLIT flag is set on the cdev, return any I/O that is larger than si_iosize_max or MAXPHYS, has more than one segment, or would have to be split because of misalignment with EFBIG. (File too large). In the event of an error, print a console message to give the user a clue about what happened. scsi_sa.c: Set the SI_NOSPLIT cdev flag on the devices created for the sa(4) driver by default. Add tunables to control whether we allow I/O splitting in physio(9). Explain in the comments that allowing I/O splitting will be deprecated for the sa(4) driver in FreeBSD 11.0. Add sysctl variables to display the maximum I/O size we can do (which could be further limited by read block limits) and the maximum I/O size that the controller can do. Limit our maximum I/O size (recorded in the cdev's si_iosize_max) by MAXPHYS. This isn't strictly necessary, because physio(9) will limit it to MAXPHYS, but it will provide some clarity for the application. Record the controller's maximum I/O size reported in the Path Inquiry CCB. sa.4: Document the block size behavior, and explain that the option of allowing physio(9) to split the I/O will disappear in FreeBSD 11.0. Sponsored by: Spectra Logic
2013-08-24 04:52:22 +00:00
if (dev->si_flags & SI_NOSPLIT) {
Fix some issues in change 254760 pointed out by Bruce Evans: - Remove excessive parenthesis - Use KNF continuation indentation - Cut down on excessive continuation lines - More consistent style in messages - Use uprintf() instead of printf() Submitted by: bde
2013-08-29 16:41:40 +00:00
uprintf("%s: request ptr %p is not "
"on a page boundary; cannot split "
Add support to physio(9) for devices that don't want I/O split and configure sa(4) to request no I/O splitting by default. For tape devices, the user needs to be able to clearly understand what blocksize is actually being used when writing to a tape device. The previous behavior of physio(9) was that it would split up any I/O that was too large for the device, or too large to fit into MAXPHYS. This means that if, for instance, the user wrote a 1MB block to a tape device, and MAXPHYS was 128KB, the 1MB write would be split into 8 128K chunks. This would be done without informing the user. This has suboptimal effects, especially when trying to communicate status to the user. In the event of an error writing to a tape (e.g. physical end of tape) in the middle of a 1MB block that has been split into 8 pieces, the user could have the first two 128K pieces written successfully, the third returned with an error, and the last 5 returned with 0 bytes written. If the user is using a standard write(2) system call, all he will see is the ENOSPC error. He won't have a clue how much actually got written. (With a writev(2) system call, he should be able to determine how much got written in addition to the error.) The solution is to prevent physio(9) from splitting the I/O. The new cdev flag, SI_NOSPLIT, tells physio that the driver does not want I/O to be split beforehand. Although the sa(4) driver now enables SI_NOSPLIT by default, that can be disabled by two loader tunables for now. It will not be configurable starting in FreeBSD 11.0. kern.cam.sa.allow_io_split allows the user to configure I/O splitting for all sa(4) driver instances. kern.cam.sa.%d.allow_io_split allows the user to configure I/O splitting for a specific sa(4) instance. There are also now three sa(4) driver sysctl variables that let the users see some sa(4) driver values. kern.cam.sa.%d.allow_io_split shows whether I/O splitting is turned on. kern.cam.sa.%d.maxio shows the maximum I/O size allowed by kernel configuration parameters (e.g. MAXPHYS, DFLTPHYS) and the capabilities of the controller. kern.cam.sa.%d.cpi_maxio shows the maximum I/O size supported by the controller. Note that a better long term solution would be to implement support for chaining buffers, so that that MAXPHYS is no longer a limiting factor for I/O size to tape and disk devices. At that point, the controller and the tape drive would become the limiting factors. sys/conf.h: Add a new cdev flag, SI_NOSPLIT, that allows a driver to tell physio not to split up I/O. sys/param.h: Bump __FreeBSD_version to 1000049 for the addition of the SI_NOSPLIT cdev flag. kern_physio.c: If the SI_NOSPLIT flag is set on the cdev, return any I/O that is larger than si_iosize_max or MAXPHYS, has more than one segment, or would have to be split because of misalignment with EFBIG. (File too large). In the event of an error, print a console message to give the user a clue about what happened. scsi_sa.c: Set the SI_NOSPLIT cdev flag on the devices created for the sa(4) driver by default. Add tunables to control whether we allow I/O splitting in physio(9). Explain in the comments that allowing I/O splitting will be deprecated for the sa(4) driver in FreeBSD 11.0. Add sysctl variables to display the maximum I/O size we can do (which could be further limited by read block limits) and the maximum I/O size that the controller can do. Limit our maximum I/O size (recorded in the cdev's si_iosize_max) by MAXPHYS. This isn't strictly necessary, because physio(9) will limit it to MAXPHYS, but it will provide some clarity for the application. Record the controller's maximum I/O size reported in the Path Inquiry CCB. sa.4: Document the block size behavior, and explain that the option of allowing physio(9) to split the I/O will disappear in FreeBSD 11.0. Sponsored by: Spectra Logic
2013-08-24 04:52:22 +00:00
"request\n", devtoname(dev),
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
bp->bio_data);
Add support to physio(9) for devices that don't want I/O split and configure sa(4) to request no I/O splitting by default. For tape devices, the user needs to be able to clearly understand what blocksize is actually being used when writing to a tape device. The previous behavior of physio(9) was that it would split up any I/O that was too large for the device, or too large to fit into MAXPHYS. This means that if, for instance, the user wrote a 1MB block to a tape device, and MAXPHYS was 128KB, the 1MB write would be split into 8 128K chunks. This would be done without informing the user. This has suboptimal effects, especially when trying to communicate status to the user. In the event of an error writing to a tape (e.g. physical end of tape) in the middle of a 1MB block that has been split into 8 pieces, the user could have the first two 128K pieces written successfully, the third returned with an error, and the last 5 returned with 0 bytes written. If the user is using a standard write(2) system call, all he will see is the ENOSPC error. He won't have a clue how much actually got written. (With a writev(2) system call, he should be able to determine how much got written in addition to the error.) The solution is to prevent physio(9) from splitting the I/O. The new cdev flag, SI_NOSPLIT, tells physio that the driver does not want I/O to be split beforehand. Although the sa(4) driver now enables SI_NOSPLIT by default, that can be disabled by two loader tunables for now. It will not be configurable starting in FreeBSD 11.0. kern.cam.sa.allow_io_split allows the user to configure I/O splitting for all sa(4) driver instances. kern.cam.sa.%d.allow_io_split allows the user to configure I/O splitting for a specific sa(4) instance. There are also now three sa(4) driver sysctl variables that let the users see some sa(4) driver values. kern.cam.sa.%d.allow_io_split shows whether I/O splitting is turned on. kern.cam.sa.%d.maxio shows the maximum I/O size allowed by kernel configuration parameters (e.g. MAXPHYS, DFLTPHYS) and the capabilities of the controller. kern.cam.sa.%d.cpi_maxio shows the maximum I/O size supported by the controller. Note that a better long term solution would be to implement support for chaining buffers, so that that MAXPHYS is no longer a limiting factor for I/O size to tape and disk devices. At that point, the controller and the tape drive would become the limiting factors. sys/conf.h: Add a new cdev flag, SI_NOSPLIT, that allows a driver to tell physio not to split up I/O. sys/param.h: Bump __FreeBSD_version to 1000049 for the addition of the SI_NOSPLIT cdev flag. kern_physio.c: If the SI_NOSPLIT flag is set on the cdev, return any I/O that is larger than si_iosize_max or MAXPHYS, has more than one segment, or would have to be split because of misalignment with EFBIG. (File too large). In the event of an error, print a console message to give the user a clue about what happened. scsi_sa.c: Set the SI_NOSPLIT cdev flag on the devices created for the sa(4) driver by default. Add tunables to control whether we allow I/O splitting in physio(9). Explain in the comments that allowing I/O splitting will be deprecated for the sa(4) driver in FreeBSD 11.0. Add sysctl variables to display the maximum I/O size we can do (which could be further limited by read block limits) and the maximum I/O size that the controller can do. Limit our maximum I/O size (recorded in the cdev's si_iosize_max) by MAXPHYS. This isn't strictly necessary, because physio(9) will limit it to MAXPHYS, but it will provide some clarity for the application. Record the controller's maximum I/O size reported in the Path Inquiry CCB. sa.4: Document the block size behavior, and explain that the option of allowing physio(9) to split the I/O will disappear in FreeBSD 11.0. Sponsored by: Spectra Logic
2013-08-24 04:52:22 +00:00
error = EFBIG;
goto doerror;
}
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
bp->bio_length = pbuf->b_kvasize;
if (poff != 0)
bp->bio_length -= PAGE_SIZE;
Give physio a makeover. - Let physio take read/write compatible args and have it use uio->uio_rw to determine the direction. - physread/physwrite are now #defines for physio - Remove the inversly named minphys(), dev->si_iosize_max takes over. - Physio() always uses pbufs. - Fix the check for non page-aligned transfers, now only unaligned transfers larger than (MAXPHYS - PAGE_SIZE) get fragmented (only interesting for tapes using max blocksize). - General wash-and-clean of code. Constructive input from: bde
1999-10-09 19:44:32 +00:00
}
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
bp->bio_bcount = bp->bio_length;
bp->bio_dev = dev;
Remove trailing whitespace.
1995-05-30 08:16:23 +00:00
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
if (pages) {
if ((npages = vm_fault_quick_hold_pages(
&curproc->p_vmspace->vm_map,
(vm_offset_t)bp->bio_data, bp->bio_length,
prot, pages, maxpages)) < 0) {
Close the remaining user address mapping races for physical I/O, CAM, and AIO. Still TODO: streamline useracc() checks. Reviewed by: alc, tegge MFC after: 7 days
2003-01-20 17:46:48 +00:00
error = EFAULT;
goto doerror;
}
physio: avoid uninitialized variables
2018-05-19 04:09:58 +00:00
if (pbuf && sa) {
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
pmap_qenter((vm_offset_t)sa,
pages, npages);
bp->bio_data = sa + poff;
} else {
bp->bio_ma = pages;
bp->bio_ma_n = npages;
bp->bio_ma_offset = poff;
bp->bio_data = unmapped_buf;
bp->bio_flags |= BIO_UNMAPPED;
}
Do not pass unmapped buffers to drivers that cannot handle them In physio, check if device can handle unmapped IO and pass an appropriately mapped buffer to the driver strategy routine. The only driver in the tree that can handle unmapped buffers is one exposed by GEOM, so mark it as such with the new flag in the driver cdevsw structure. This fixes insta-panics on hosts, running dconschat, as /dev/fwmem is an example of the driver that makes use of physio routine, but bypasses the g_down thread, where the buffer gets mapped normally. Discussed with: kib (earlier version)
2013-03-26 01:17:06 +00:00
}
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
Add missing NULL check in physio(). When destroying a character device the si_devsw field is set to NULL before all references are gone, to indicate the character device is going away. This can cause a NULL-dereference fault inside physio(). The callers of physio() should own a thread reference on the cdev and if si_devsw is seen as non-NULL, it is usable during the execution of the function. Else an ENXIO error code is returned. Reviewed by: kib MFC after: 2 weeks
2015-10-29 13:53:37 +00:00
csw->d_strategy(bp);
- Add a lock for protecting against msleep(bp, ...) wakeup(bp) races. - Create a new function bdone() which sets B_DONE and calls wakup(bp). This is suitable for use as b_iodone for buf consumers who are not going through the buf cache. - Create a new function bwait() which waits for the buf to be done at a set priority and with a specific wmesg. - Replace several cases where the above functionality was implemented without locking with the new functions.
2003-03-13 07:31:45 +00:00
if (uio->uio_rw == UIO_READ)
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
biowait(bp, "physrd");
- Add a lock for protecting against msleep(bp, ...) wakeup(bp) races. - Create a new function bdone() which sets B_DONE and calls wakup(bp). This is suitable for use as b_iodone for buf consumers who are not going through the buf cache. - Create a new function bwait() which waits for the buf to be done at a set priority and with a specific wmesg. - Replace several cases where the above functionality was implemented without locking with the new functions.
2003-03-13 07:31:45 +00:00
else
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
biowait(bp, "physwr");
if (pages) {
if (pbuf)
pmap_qremove((vm_offset_t)sa, npages);
vm_page_unhold_pages(pages, npages);
}
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
iolen = bp->bio_length - bp->bio_resid;
if (iolen == 0 && !(bp->bio_flags & BIO_ERROR))
Give physio a makeover. - Let physio take read/write compatible args and have it use uio->uio_rw to determine the direction. - physread/physwrite are now #defines for physio - Remove the inversly named minphys(), dev->si_iosize_max takes over. - Physio() always uses pbufs. - Fix the check for non page-aligned transfers, now only unaligned transfers larger than (MAXPHYS - PAGE_SIZE) get fragmented (only interesting for tapes using max blocksize). - General wash-and-clean of code. Constructive input from: bde
1999-10-09 19:44:32 +00:00
goto doerror; /* EOF */
uio->uio_iov[i].iov_len -= iolen;
Change iov_base's type from `char *' to the standard `void *'. All uses of iov_base which assume its type is `char *' (in order to do pointer arithmetic) have been updated to cast iov_base to `char *'.
2002-10-11 14:58:34 +00:00
uio->uio_iov[i].iov_base =
(char *)uio->uio_iov[i].iov_base + iolen;
Give physio a makeover. - Let physio take read/write compatible args and have it use uio->uio_rw to determine the direction. - physread/physwrite are now #defines for physio - Remove the inversly named minphys(), dev->si_iosize_max takes over. - Physio() always uses pbufs. - Fix the check for non page-aligned transfers, now only unaligned transfers larger than (MAXPHYS - PAGE_SIZE) get fragmented (only interesting for tapes using max blocksize). - General wash-and-clean of code. Constructive input from: bde
1999-10-09 19:44:32 +00:00
uio->uio_resid -= iolen;
uio->uio_offset += iolen;
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
if (bp->bio_flags & BIO_ERROR) {
error = bp->bio_error;
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
goto doerror;
}
}
}
doerror:
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
if (pbuf)
Allocate pager bufs from UMA instead of 80-ish mutex protected linked list. o In vm_pager_bufferinit() create pbuf_zone and start accounting on how many pbufs are we going to have set. In various subsystems that are going to utilize pbufs create private zones via call to pbuf_zsecond_create(). The latter calls uma_zsecond_create(), and sets a limit on created zone. After startup preallocate pbufs according to requirements of all pbuf zones. Subsystems that used to have a private limit with old allocator now have private pbuf zones: md(4), fusefs, NFS client, smbfs, VFS cluster, FFS, swap, vnode pager. The following subsystems use shared pbuf zone: cam(4), nvme(4), physio(9), aio(4). They should have their private limits, but changing that is out of scope of this commit. o Fetch tunable value of kern.nswbuf from init_param2() and while here move NSWBUF_MIN to opt_param.h and eliminate opt_swap.h, that was holding only this option. Default values aren't touched by this commit, but they probably should be reviewed wrt to modern hardware. This change removes a tight bottleneck from sendfile(2) operation, that uses pbufs in vnode pager. Other pagers also would benefit from faster allocation. Together with: gallatin Tested by: pho
2019-01-15 01:02:16 +00:00
uma_zfree(pbuf_zone, pbuf);
Rewrite physio() to not allocate pbufs for unmapped I/O. pbufs is a limited resource, and their allocator is not SMP-scalable. So instead of always allocating pbuf to immediately convert it to bio, allocate bio just here. If buffer needs kernel mapping, then pbuf is still allocated, but used only as a source of KVA and storage for a list of held pages. On 40-core system doing many 512-byte reads from user level to array of raw SSDs this change removes huge lock congestion inside pbuf allocator. It improves peak performance from ~300K to ~1.2M IOPS. On my previous 24-core system this problem also existed, but was less serious. Reviewed by: kib MFC after: 2 weeks
2015-04-21 10:55:53 +00:00
else if (pages)
free(pages, M_DEVBUF);
g_destroy_bio(bp);
Use the reference counted PHOLD()/PRELE() rather than P_PHYSIO.
1999-04-06 03:04:47 +00:00
PRELE(curproc);
The big 4.4BSD Lite to FreeBSD 2.0.0 (Development) patch. Reviewed by: Rodney W. Grimes Submitted by: John Dyson and David Greenman
1994-05-25 09:21:21 +00:00
return (error);
BSD 4.4 Lite Kernel Sources
1994-05-24 10:09:53 +00:00
}
Reference in New Issue Copy Permalink