d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
93cbefc174
freebsd-skq/sys/geom/geom_io.c

964 lines
25 KiB
C
Raw Normal View History

First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
/*-
* Copyright (c) 2002 Poul-Henning Kamp
* Copyright (c) 2002 Networks Associates Technology, Inc.
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
* Copyright (c) 2013 The FreeBSD Foundation
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
* All rights reserved.
*
* This software was developed for the FreeBSD Project by Poul-Henning Kamp
* and NAI Labs, the Security Research Division of Network Associates, Inc.
* under DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the
* DARPA CHATS research program.
*
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
* Portions of this software were developed by Konstantin Belousov
* under sponsorship from the FreeBSD Foundation.
*
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. The names of the authors may not be used to endorse or promote
* products derived from this software without specific prior written
* permission.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
Use __FBSDID(). Approved by: phk
2003-06-11 06:49:16 +00:00
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/bio.h>
Add KTR_GEOM, which allows tracing of basic GEOM I/O events occuring in the g_up and g_down threads. Each time a bio is propelled up and down the stack, an event is generating showing the provider, offset, and length, as well as thread wakeup and work status information.
2004-10-21 18:35:24 +00:00
#include <sys/ktr.h>
- Add a new simple facility for marking the current thread as being in a state where sleeping on a sleep queue is not allowed. The facility doesn't support recursion but uses a simple private per-thread flag (TDP_NOSLEEPING). The sleepq_add() function will panic if the flag is set and INVARIANTS is enabled. - Use this new facility to replace the g_xup and g_xdown mutexes that were (ab)used to achieve similar behavior. - Disallow sleeping in interrupt threads when invoking interrupt handlers. MFC after: 1 week Reviewed by: phk
2005-09-15 19:05:37 +00:00
#include <sys/proc.h>
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
#include <sys/stack.h>
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
#include <sys/sysctl.h>
- Add a general purpose resource allocator, vmem, from NetBSD. It was originally inspired by the Solaris vmem detailed in the proceedings of usenix 2001. The NetBSD version was heavily refactored for bugs and simplicity. - Use this resource allocator to allocate the buffer and transient maps. Buffer cache defrags are reduced by 25% when used by filesystems with mixed block sizes. Ultimately this may permit dynamic buffer cache sizing on low KVA machines. Discussed with: alc, kib, attilio Tested by: pho Sponsored by: EMC / Isilon Storage Division
2013-06-28 03:51:20 +00:00
#include <sys/vmem.h>
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
#include <sys/errno.h>
#include <geom/geom.h>
Eliminate some thread pointers which do not make sense anymore. Split private parts of geom.h into geom_int.h. The latter should never be included in class implemtations.
2002-03-26 22:07:38 +00:00
#include <geom/geom_int.h>
Retire the GEOM private statistics code and use devstat instead.
2003-03-18 09:42:33 +00:00
#include <sys/devicestat.h>
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
Use an uma-zone for allocation bio requests.
2003-05-02 12:36:12 +00:00
#include <vm/uma.h>
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
#include <vm/vm.h>
#include <vm/vm_param.h>
#include <vm/vm_kern.h>
#include <vm/vm_page.h>
#include <vm/vm_object.h>
#include <vm/vm_extern.h>
#include <vm/vm_map.h>
Use an uma-zone for allocation bio requests.
2003-05-02 12:36:12 +00:00
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
static int g_io_transient_map_bio(struct bio *bp);
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
static struct g_bioq g_bio_run_down;
static struct g_bioq g_bio_run_up;
Bring back the geom_bioqueues, they _are_ a good idea. ATA will uses these RSN.
2004-01-28 08:39:18 +00:00
static struct g_bioq g_bio_run_task;
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
malloc(9) with M_NOWAIT seems to return NULL a lot more than I would have expected under -current. This is a problem for GEOM because the up/down threads cannot sleep waiting for memory to become free. The reason they cannot sleep is that paging things out to disk may be the only way we can clear up some RAM. Nice catch-22 there. Implement a rudimentary ENOMEM recovery strategy: If an I/O request fails with an error code of ENOMEM, schedule it for a retry, and tell the down-thread to sleep hz/10 to get other parts of the system a chance to free up some memory, in particular the up-path in GEOM. All caches should probably start to monitor malloc(9) failures using the new malloc_last_fail() function, and release when it indicates congestion. Sponsored by: DARPA & NAI Labs.
2002-11-02 11:08:07 +00:00
static u_int pace;
Use an uma-zone for allocation bio requests.
2003-05-02 12:36:12 +00:00
static uma_zone_t biozone;
malloc(9) with M_NOWAIT seems to return NULL a lot more than I would have expected under -current. This is a problem for GEOM because the up/down threads cannot sleep waiting for memory to become free. The reason they cannot sleep is that paging things out to disk may be the only way we can clear up some RAM. Nice catch-22 there. Implement a rudimentary ENOMEM recovery strategy: If an I/O request fails with an error code of ENOMEM, schedule it for a retry, and tell the down-thread to sleep hz/10 to get other parts of the system a chance to free up some memory, in particular the up-path in GEOM. All caches should probably start to monitor malloc(9) failures using the new malloc_last_fail() function, and release when it indicates congestion. Sponsored by: DARPA & NAI Labs.
2002-11-02 11:08:07 +00:00
As discussed in the devsummit, introduce two fields in the struct bio to store classification information, and a hook for classifier functions that can be called by g_io_request(). This code is from Fabio Checconi as part of his GSOC work.
2009-06-11 09:55:26 +00:00
/*
* The head of the list of classifiers used in g_io_request.
* Use g_register_classifier() and g_unregister_classifier()
* to add/remove entries to the list.
* Classifiers are invoked in registration order.
*/
static TAILQ_HEAD(g_classifier_tailq, g_classifier_hook)
g_classifier_tailq = TAILQ_HEAD_INITIALIZER(g_classifier_tailq);
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
#include <machine/atomic.h>
static void
g_bioq_lock(struct g_bioq *bq)
{
mtx_lock(&bq->bio_queue_lock);
}
static void
g_bioq_unlock(struct g_bioq *bq)
{
mtx_unlock(&bq->bio_queue_lock);
}
#if 0
static void
g_bioq_destroy(struct g_bioq *bq)
{
mtx_destroy(&bq->bio_queue_lock);
}
#endif
static void
g_bioq_init(struct g_bioq *bq)
{
TAILQ_INIT(&bq->bio_queue);
Change callers of mtx_init() to pass in an appropriate lock type name. In most cases NULL is passed, but in some cases such as network driver locks (which use the MTX_NETWORK_LOCK macro) and UMA zone locks, a name is used. Tested on: i386, alpha, sparc64
2002-04-04 21:03:38 +00:00
mtx_init(&bq->bio_queue_lock, "bio queue", NULL, MTX_DEF);
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
}
static struct bio *
g_bioq_first(struct g_bioq *bq)
{
struct bio *bp;
bp = TAILQ_FIRST(&bq->bio_queue);
if (bp != NULL) {
Add more KASSERTS and checks.
2004-08-30 09:33:06 +00:00
KASSERT((bp->bio_flags & BIO_ONQUEUE),
("Bio not on queue bp=%p target %p", bp, bq));
bp->bio_flags &= ~BIO_ONQUEUE;
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
TAILQ_REMOVE(&bq->bio_queue, bp, bio_queue);
bq->bio_queue_length--;
}
return (bp);
}
struct bio *
g_new_bio(void)
{
struct bio *bp;
Use an uma-zone for allocation bio requests.
2003-05-02 12:36:12 +00:00
bp = uma_zalloc(biozone, M_NOWAIT | M_ZERO);
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
#ifdef KTR
Save stack only when KTR_GEOM is both compiled into the kernel and enabled in debug.ktr.mask. Because saving stack is very expensive, it's better only to do it when one really wants to. Reported by: Dan Nelson
2007-10-26 06:55:00 +00:00
if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) {
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
struct stack st;
CTR1(KTR_GEOM, "g_new_bio(): %p", bp);
stack_save(&st);
CTRSTACK(KTR_GEOM, &st, 3, 0);
}
#endif
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
return (bp);
}
Introduce g_alloc_bio() as a waiting variant of g_new_bio(). Use in places where we can sleep and where we previously failed to check for a NULL pointer. MT5 candidate.
2004-08-27 14:43:11 +00:00
struct bio *
g_alloc_bio(void)
{
struct bio *bp;
bp = uma_zalloc(biozone, M_WAITOK | M_ZERO);
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
#ifdef KTR
Save stack only when KTR_GEOM is both compiled into the kernel and enabled in debug.ktr.mask. Because saving stack is very expensive, it's better only to do it when one really wants to. Reported by: Dan Nelson
2007-10-26 06:55:00 +00:00
if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) {
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
struct stack st;
CTR1(KTR_GEOM, "g_alloc_bio(): %p", bp);
stack_save(&st);
CTRSTACK(KTR_GEOM, &st, 3, 0);
}
#endif
Introduce g_alloc_bio() as a waiting variant of g_new_bio(). Use in places where we can sleep and where we previously failed to check for a NULL pointer. MT5 candidate.
2004-08-27 14:43:11 +00:00
return (bp);
}
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
void
g_destroy_bio(struct bio *bp)
{
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
#ifdef KTR
Save stack only when KTR_GEOM is both compiled into the kernel and enabled in debug.ktr.mask. Because saving stack is very expensive, it's better only to do it when one really wants to. Reported by: Dan Nelson
2007-10-26 06:55:00 +00:00
if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) {
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
struct stack st;
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
CTR1(KTR_GEOM, "g_destroy_bio(): %p", bp);
stack_save(&st);
CTRSTACK(KTR_GEOM, &st, 3, 0);
}
#endif
Use an uma-zone for allocation bio requests.
2003-05-02 12:36:12 +00:00
uma_zfree(biozone, bp);
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
}
struct bio *
g_clone_bio(struct bio *bp)
{
struct bio *bp2;
Use an uma-zone for allocation bio requests.
2003-05-02 12:36:12 +00:00
bp2 = uma_zalloc(biozone, M_NOWAIT | M_ZERO);
Allocate bio's with M_NOWAIT and let the caller deal with the problems. Sponsored by: DARPA & NAI Labs.
2002-09-27 20:53:47 +00:00
if (bp2 != NULL) {
Rename bio_linkage to the more obvious bio_parent. Add bio_t0 timestamp, and include <sys/time.h> where needed
2003-02-07 21:09:51 +00:00
bp2->bio_parent = bp;
Allocate bio's with M_NOWAIT and let the caller deal with the problems. Sponsored by: DARPA & NAI Labs.
2002-09-27 20:53:47 +00:00
bp2->bio_cmd = bp->bio_cmd;
Clone BIO_ORDERED flag, for disk drivers (namely CAM) that try to consume it. Sponsored by: Intel Discussed with: gibbs, scottl
2012-08-07 20:16:10 +00:00
/*
* BIO_ORDERED flag may be used by disk drivers to enforce
* ordering restrictions, so this flag needs to be cloned.
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
* BIO_UNMAPPED should be inherited, to properly indicate
* which way the buffer is passed.
Clone BIO_ORDERED flag, for disk drivers (namely CAM) that try to consume it. Sponsored by: Intel Discussed with: gibbs, scottl
2012-08-07 20:16:10 +00:00
* Other bio flags are not suitable for cloning.
*/
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
bp2->bio_flags = bp->bio_flags & (BIO_ORDERED | BIO_UNMAPPED);
Allocate bio's with M_NOWAIT and let the caller deal with the problems. Sponsored by: DARPA & NAI Labs.
2002-09-27 20:53:47 +00:00
bp2->bio_length = bp->bio_length;
bp2->bio_offset = bp->bio_offset;
bp2->bio_data = bp->bio_data;
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
bp2->bio_ma = bp->bio_ma;
bp2->bio_ma_n = bp->bio_ma_n;
bp2->bio_ma_offset = bp->bio_ma_offset;
Allocate bio's with M_NOWAIT and let the caller deal with the problems. Sponsored by: DARPA & NAI Labs.
2002-09-27 20:53:47 +00:00
bp2->bio_attribute = bp->bio_attribute;
As discussed in the devsummit, introduce two fields in the struct bio to store classification information, and a hook for classifier functions that can be called by g_io_request(). This code is from Fabio Checconi as part of his GSOC work.
2009-06-11 09:55:26 +00:00
/* Inherit classification info from the parent */
bp2->bio_classifier1 = bp->bio_classifier1;
bp2->bio_classifier2 = bp->bio_classifier2;
Commit the correct copy of the g_stat structure. Add debug.sizeof.g_stat sysctl. Set the id field of the g_stat when we create consumers and providers. Remove biocount from consumer, we will use the counters in the g_stat structure instead. Replace one field which will need to be atomically manipulated with two fields which will not (stat.nop and stat.nend). Change add companion field to bio_children: bio_inbed for the exact same reason. Don't output the biocount in the confdot output. Fix KASSERT in g_io_request(). Add sysctl kern.geom.collectstats defaulting to off. Collect the following raw statistics conditioned on this sysctl: for each consumer and provider { total number of operations started. total number of operations completed. time last operation completed. sum of idle-time. for each of BIO_READ, BIO_WRITE and BIO_DELETE { number of operations completed. number of bytes completed. number of ENOMEM errors. number of other errors. sum of transaction time. } } API for getting hold of these statistics data not included yet.
2003-02-07 23:08:24 +00:00
bp->bio_children++;
Allocate bio's with M_NOWAIT and let the caller deal with the problems. Sponsored by: DARPA & NAI Labs.
2002-09-27 20:53:47 +00:00
}
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
#ifdef KTR
Save stack only when KTR_GEOM is both compiled into the kernel and enabled in debug.ktr.mask. Because saving stack is very expensive, it's better only to do it when one really wants to. Reported by: Dan Nelson
2007-10-26 06:55:00 +00:00
if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) {
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
struct stack st;
Fix a typo.
2006-03-13 14:59:57 +00:00
CTR2(KTR_GEOM, "g_clone_bio(%p): %p", bp, bp2);
Use KTR to log allocations and destructions of bios. This should hopefully allow to track down "duplicate free of g_bio" panics.
2005-08-29 11:39:24 +00:00
stack_save(&st);
CTRSTACK(KTR_GEOM, &st, 3, 0);
}
#endif
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
return(bp2);
}
Add g_duplicate_bio() function which does the same thing what g_clone_bio() is doing, but g_duplicate_bio() allocates new bio with M_WAITOK flag.
2006-06-05 21:13:22 +00:00
struct bio *
g_duplicate_bio(struct bio *bp)
{
struct bio *bp2;
bp2 = uma_zalloc(biozone, M_WAITOK | M_ZERO);
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
bp2->bio_flags = bp->bio_flags & BIO_UNMAPPED;
Add g_duplicate_bio() function which does the same thing what g_clone_bio() is doing, but g_duplicate_bio() allocates new bio with M_WAITOK flag.
2006-06-05 21:13:22 +00:00
bp2->bio_parent = bp;
bp2->bio_cmd = bp->bio_cmd;
bp2->bio_length = bp->bio_length;
bp2->bio_offset = bp->bio_offset;
bp2->bio_data = bp->bio_data;
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
bp2->bio_ma = bp->bio_ma;
bp2->bio_ma_n = bp->bio_ma_n;
bp2->bio_ma_offset = bp->bio_ma_offset;
Add g_duplicate_bio() function which does the same thing what g_clone_bio() is doing, but g_duplicate_bio() allocates new bio with M_WAITOK flag.
2006-06-05 21:13:22 +00:00
bp2->bio_attribute = bp->bio_attribute;
bp->bio_children++;
#ifdef KTR
Save stack only when KTR_GEOM is both compiled into the kernel and enabled in debug.ktr.mask. Because saving stack is very expensive, it's better only to do it when one really wants to. Reported by: Dan Nelson
2007-10-26 06:55:00 +00:00
if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) {
Add g_duplicate_bio() function which does the same thing what g_clone_bio() is doing, but g_duplicate_bio() allocates new bio with M_WAITOK flag.
2006-06-05 21:13:22 +00:00
struct stack st;
CTR2(KTR_GEOM, "g_duplicate_bio(%p): %p", bp, bp2);
stack_save(&st);
CTRSTACK(KTR_GEOM, &st, 3, 0);
}
#endif
return(bp2);
}
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
void
g_io_init()
{
g_bioq_init(&g_bio_run_down);
g_bioq_init(&g_bio_run_up);
Bring back the geom_bioqueues, they _are_ a good idea. ATA will uses these RSN.
2004-01-28 08:39:18 +00:00
g_bioq_init(&g_bio_run_task);
Use an uma-zone for allocation bio requests.
2003-05-02 12:36:12 +00:00
biozone = uma_zcreate("g_bio", sizeof (struct bio),
NULL, NULL,
NULL, NULL,
0, 0);
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
}
int
Constifixation of attribute argument to g_io_[gs]etattr() Sponsored by: DARPA & NAI Labs
2002-04-09 15:12:05 +00:00
g_io_getattr(const char *attr, struct g_consumer *cp, int *len, void *ptr)
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
{
struct bio *bp;
int error;
g_trace(G_T_BIO, "bio_getattr(%s)", attr);
Introduce g_alloc_bio() as a waiting variant of g_new_bio(). Use in places where we can sleep and where we previously failed to check for a NULL pointer. MT5 candidate.
2004-08-27 14:43:11 +00:00
bp = g_alloc_bio();
For now, don't wait for drives to stop returning EBUSY. There is too much broken harware around it seems. Sponsored by: DARPA & NAI Labs.
2002-10-08 07:03:58 +00:00
bp->bio_cmd = BIO_GETATTR;
bp->bio_done = NULL;
bp->bio_attribute = attr;
bp->bio_length = *len;
bp->bio_data = ptr;
g_io_request(bp, cp);
error = biowait(bp, "ggetattr");
*len = bp->bio_completed;
g_destroy_bio(bp);
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
return (error);
}
Add a new I/O request - BIO_FLUSH, which basically tells providers below to flush their caches. For now will mostly be used by disks to flush their write cache. Sponsored by: home.pl
2006-10-31 21:11:21 +00:00
int
g_io_flush(struct g_consumer *cp)
{
struct bio *bp;
int error;
g_trace(G_T_BIO, "bio_flush(%s)", cp->provider->name);
bp = g_alloc_bio();
bp->bio_cmd = BIO_FLUSH;
Correct bioq_disksort so that bioq_insert_tail() offers barrier semantic. Add the BIO_ORDERED flag for struct bio and update bio clients to use it. The barrier semantics of bioq_insert_tail() were broken in two ways: o In bioq_disksort(), an added bio could be inserted at the head of the queue, even when a barrier was present, if the sort key for the new entry was less than that of the last queued barrier bio. o The last_offset used to generate the sort key for newly queued bios did not stay at the position of the barrier until either the barrier was de-queued, or a new barrier (which updates last_offset) was queued. When a barrier is in effect, we know that the disk will pass through the barrier position just before the "blocked bios" are released, so using the barrier's offset for last_offset is the optimal choice. sys/geom/sched/subr_disk.c: sys/kern/subr_disk.c: o Update last_offset in bioq_insert_tail(). o Only update last_offset in bioq_remove() if the removed bio is at the head of the queue (typically due to a call via bioq_takefirst()) and no barrier is active. o In bioq_disksort(), if we have a barrier (insert_point is non-NULL), set prev to the barrier and cur to it's next element. Now that last_offset is kept at the barrier position, this change isn't strictly necessary, but since we have to take a decision branch anyway, it does avoid one, no-op, loop iteration in the while loop that immediately follows. o In bioq_disksort(), bypass the normal sort for bios with the BIO_ORDERED attribute and instead insert them into the queue with bioq_insert_tail(). bioq_insert_tail() not only gives the desired command order during insertion, but also provides barrier semantics so that commands disksorted in the future cannot pass the just enqueued transaction. sys/sys/bio.h: Add BIO_ORDERED as bit 4 of the bio_flags field in struct bio. sys/cam/ata/ata_da.c: sys/cam/scsi/scsi_da.c Use an ordered command for SCSI/ATA-NCQ commands issued in response to bios with the BIO_ORDERED flag set. sys/cam/scsi/scsi_da.c Use an ordered tag when issuing a synchronize cache command. Wrap some lines to 80 columns. sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c sys/geom/geom_io.c Mark bios with the BIO_FLUSH command as BIO_ORDERED. Sponsored by: Spectra Logic Corporation MFC after: 1 month
2010-09-02 19:40:28 +00:00
bp->bio_flags |= BIO_ORDERED;
Add a new I/O request - BIO_FLUSH, which basically tells providers below to flush their caches. For now will mostly be used by disks to flush their write cache. Sponsored by: home.pl
2006-10-31 21:11:21 +00:00
bp->bio_done = NULL;
bp->bio_attribute = NULL;
bp->bio_offset = cp->provider->mediasize;
bp->bio_length = 0;
bp->bio_data = NULL;
g_io_request(bp, cp);
error = biowait(bp, "gflush");
g_destroy_bio(bp);
return (error);
}
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
static int
g_io_check(struct bio *bp)
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
{
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
struct g_consumer *cp;
struct g_provider *pp;
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
off_t excess;
int error;
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
cp = bp->bio_from;
pp = bp->bio_to;
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
/* Fail if access counters dont allow the operation */
Centralize EOF handling and improve access controls for bio scheduling. Sponsored by: DARPA & NAI Labs
2002-04-04 09:58:20 +00:00
switch(bp->bio_cmd) {
case BIO_READ:
case BIO_GETATTR:
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
if (cp->acr == 0)
return (EPERM);
Centralize EOF handling and improve access controls for bio scheduling. Sponsored by: DARPA & NAI Labs
2002-04-04 09:58:20 +00:00
break;
case BIO_WRITE:
case BIO_DELETE:
Add a new I/O request - BIO_FLUSH, which basically tells providers below to flush their caches. For now will mostly be used by disks to flush their write cache. Sponsored by: home.pl
2006-10-31 21:11:21 +00:00
case BIO_FLUSH:
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
if (cp->acw == 0)
return (EPERM);
Centralize EOF handling and improve access controls for bio scheduling. Sponsored by: DARPA & NAI Labs
2002-04-04 09:58:20 +00:00
break;
default:
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
return (EPERM);
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
}
Centralize EOF handling and improve access controls for bio scheduling. Sponsored by: DARPA & NAI Labs
2002-04-04 09:58:20 +00:00
/* if provider is marked for error, don't disturb. */
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
if (pp->error)
return (pp->error);
Implement media change notification for DA and CD removable media devices. It includes three parts: 1) Modifications to CAM to detect media media changes and report them to disk(9) layer. For modern SATA (and potentially UAS) devices it utilizes Asynchronous Notification mechanism to receive events from hardware. Active polling with TEST UNIT READY commands with 3 seconds period is used for incapable hardware. After that both CD and DA drivers work the same way, detecting two conditions: "NOT READY: Medium not present" after medium was detected previously, and "UNIT ATTENTION: Not ready to ready change, medium may have changed". First one reported to disk(9) as media removal, second as media insert/change. To reliably receive second event new AC_UNIT_ATTENTION async added to make UAs broadcasted to all periphs by generic error handling code in cam_periph_error(). 2) Modifications to GEOM core to handle media remove and change events. Media removal handled by spoiling all consumers attached to the provider. Media change event also schedules provider retaste after spoiling to probe new media. New flag G_CF_ORPHAN was added to consumers to reflect that consumer is in process of destruction. It allows retaste to create new geom instance of the same class, while previous one is still dying. 3) Modifications to some GEOM classes: DEV -- to report media change events to devd; VFS -- to handle spoiling same as orphan to prevent accessing replaced media. PART class already handles spoiling alike to orphan. Reviewed by: silence on geom@ and scsi@ Tested by: avg Sponsored by: iXsystems, Inc. / PC-BSD MFC after: 2 months
2012-07-29 11:51:48 +00:00
if (cp->flags & G_CF_ORPHAN)
return (ENXIO);
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
Centralize EOF handling and improve access controls for bio scheduling. Sponsored by: DARPA & NAI Labs
2002-04-04 09:58:20 +00:00
switch(bp->bio_cmd) {
case BIO_READ:
case BIO_WRITE:
case BIO_DELETE:
g_io_check: respond to zero pp->mediasize with ENXIO Previsouly this condition was reported with EIO by bio_offset > mediasize check. Perhaps that check should be extended to bio_offset+bio_length > mediasize. MFC after: 1 week
2010-04-15 08:39:56 +00:00
/* Zero sectorsize or mediasize is probably a lack of media. */
if (pp->sectorsize == 0 || pp->mediasize == 0)
Forgotten commit: If a provider has zero sectorsize, it is an indication of lack of media. Tripped up: peter
2003-10-22 06:32:20 +00:00
return (ENXIO);
Balk at unaligned requests. MFC candidate.
2002-12-18 19:53:59 +00:00
/* Reject I/O not on sector boundary */
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
if (bp->bio_offset % pp->sectorsize)
return (EINVAL);
Balk at unaligned requests. MFC candidate.
2002-12-18 19:53:59 +00:00
/* Reject I/O not integral sector long */
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
if (bp->bio_length % pp->sectorsize)
return (EINVAL);
Remove KASSERT check for negative bio_offsets, add "normal" EIO error return for same.
2003-10-19 19:06:54 +00:00
/* Reject requests before or past the end of media. */
if (bp->bio_offset < 0)
return (EIO);
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
if (bp->bio_offset > pp->mediasize)
return (EIO);
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
/* Truncate requests to the end of providers media. */
excess = bp->bio_offset + bp->bio_length;
if (excess > bp->bio_to->mediasize) {
KASSERT((bp->bio_flags & BIO_UNMAPPED) == 0 ||
round_page(bp->bio_ma_offset +
bp->bio_length) / PAGE_SIZE == bp->bio_ma_n,
("excess bio %p too short", bp));
excess -= bp->bio_to->mediasize;
bp->bio_length -= excess;
if ((bp->bio_flags & BIO_UNMAPPED) != 0) {
bp->bio_ma_n = round_page(bp->bio_ma_offset +
bp->bio_length) / PAGE_SIZE;
}
if (excess > 0)
CTR3(KTR_GEOM, "g_down truncated bio "
"%p provider %s by %d", bp,
bp->bio_to->name, excess);
}
/* Deliver zero length transfers right here. */
if (bp->bio_length == 0) {
CTR2(KTR_GEOM, "g_down terminated 0-length "
"bp %p provider %s", bp, bp->bio_to->name);
return (0);
}
if ((bp->bio_flags & BIO_UNMAPPED) != 0 &&
(bp->bio_to->flags & G_PF_ACCEPT_UNMAPPED) == 0 &&
(bp->bio_cmd == BIO_READ || bp->bio_cmd == BIO_WRITE)) {
if ((error = g_io_transient_map_bio(bp)) >= 0)
return (error);
}
Centralize EOF handling and improve access controls for bio scheduling. Sponsored by: DARPA & NAI Labs
2002-04-04 09:58:20 +00:00
break;
default:
break;
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
}
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
return (EJUSTRETURN);
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
}
As discussed in the devsummit, introduce two fields in the struct bio to store classification information, and a hook for classifier functions that can be called by g_io_request(). This code is from Fabio Checconi as part of his GSOC work.
2009-06-11 09:55:26 +00:00
/*
* bio classification support.
*
* g_register_classifier() and g_unregister_classifier()
* are used to add/remove a classifier from the list.
* The list is protected using the g_bio_run_down lock,
* because the classifiers are called in this path.
*
* g_io_request() passes bio's that are not already classified
* (i.e. those with bio_classifier1 == NULL) to g_run_classifiers().
* Classifiers can store their result in the two fields
* bio_classifier1 and bio_classifier2.
* A classifier that updates one of the fields should
* return a non-zero value.
* If no classifier updates the field, g_run_classifiers() sets
* bio_classifier1 = BIO_NOTCLASSIFIED to avoid further calls.
*/
int
g_register_classifier(struct g_classifier_hook *hook)
{
g_bioq_lock(&g_bio_run_down);
TAILQ_INSERT_TAIL(&g_classifier_tailq, hook, link);
g_bioq_unlock(&g_bio_run_down);
return (0);
}
void
g_unregister_classifier(struct g_classifier_hook *hook)
{
struct g_classifier_hook *entry;
g_bioq_lock(&g_bio_run_down);
TAILQ_FOREACH(entry, &g_classifier_tailq, link) {
if (entry == hook) {
TAILQ_REMOVE(&g_classifier_tailq, hook, link);
break;
}
}
g_bioq_unlock(&g_bio_run_down);
}
static void
g_run_classifiers(struct bio *bp)
{
struct g_classifier_hook *hook;
int classified = 0;
TAILQ_FOREACH(hook, &g_classifier_tailq, link)
classified |= hook->func(hook->arg, bp);
if (!classified)
bp->bio_classifier1 = BIO_NOTCLASSIFIED;
}
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
void
g_io_request(struct bio *bp, struct g_consumer *cp)
{
Commit the correct copy of the g_stat structure. Add debug.sizeof.g_stat sysctl. Set the id field of the g_stat when we create consumers and providers. Remove biocount from consumer, we will use the counters in the g_stat structure instead. Replace one field which will need to be atomically manipulated with two fields which will not (stat.nop and stat.nend). Change add companion field to bio_children: bio_inbed for the exact same reason. Don't output the biocount in the confdot output. Fix KASSERT in g_io_request(). Add sysctl kern.geom.collectstats defaulting to off. Collect the following raw statistics conditioned on this sysctl: for each consumer and provider { total number of operations started. total number of operations completed. time last operation completed. sum of idle-time. for each of BIO_READ, BIO_WRITE and BIO_DELETE { number of operations completed. number of bytes completed. number of ENOMEM errors. number of other errors. sum of transaction time. } } API for getting hold of these statistics data not included yet.
2003-02-07 23:08:24 +00:00
struct g_provider *pp;
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
struct mtx *mtxp;
int direct, error, first;
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
KASSERT(cp != NULL, ("NULL cp in g_io_request"));
KASSERT(bp != NULL, ("NULL bp in g_io_request"));
Reorder a couple of KASSERTS to give more sensible messages. Found by: GEOM 101 class of '03
2003-09-11 00:49:02 +00:00
pp = cp->provider;
Commit the correct copy of the g_stat structure. Add debug.sizeof.g_stat sysctl. Set the id field of the g_stat when we create consumers and providers. Remove biocount from consumer, we will use the counters in the g_stat structure instead. Replace one field which will need to be atomically manipulated with two fields which will not (stat.nop and stat.nend). Change add companion field to bio_children: bio_inbed for the exact same reason. Don't output the biocount in the confdot output. Fix KASSERT in g_io_request(). Add sysctl kern.geom.collectstats defaulting to off. Collect the following raw statistics conditioned on this sysctl: for each consumer and provider { total number of operations started. total number of operations completed. time last operation completed. sum of idle-time. for each of BIO_READ, BIO_WRITE and BIO_DELETE { number of operations completed. number of bytes completed. number of ENOMEM errors. number of other errors. sum of transaction time. } } API for getting hold of these statistics data not included yet.
2003-02-07 23:08:24 +00:00
KASSERT(pp != NULL, ("consumer not attached in g_io_request"));
Assert proper use of bio_caller1, bio_caller2, bio_cflags, bio_driver1, bio_driver2 and bio_pflags fields. Reviewed by: phk
2006-03-01 19:01:58 +00:00
#ifdef DIAGNOSTIC
KASSERT(bp->bio_driver1 == NULL,
("bio_driver1 used by the consumer (geom %s)", cp->geom->name));
KASSERT(bp->bio_driver2 == NULL,
("bio_driver2 used by the consumer (geom %s)", cp->geom->name));
KASSERT(bp->bio_pflags == 0,
("bio_pflags used by the consumer (geom %s)", cp->geom->name));
/*
* Remember consumer's private fields, so we can detect if they were
* modified by the provider.
*/
bp->_bio_caller1 = bp->bio_caller1;
bp->_bio_caller2 = bp->bio_caller2;
bp->_bio_cflags = bp->bio_cflags;
#endif
Commit the correct copy of the g_stat structure. Add debug.sizeof.g_stat sysctl. Set the id field of the g_stat when we create consumers and providers. Remove biocount from consumer, we will use the counters in the g_stat structure instead. Replace one field which will need to be atomically manipulated with two fields which will not (stat.nop and stat.nend). Change add companion field to bio_children: bio_inbed for the exact same reason. Don't output the biocount in the confdot output. Fix KASSERT in g_io_request(). Add sysctl kern.geom.collectstats defaulting to off. Collect the following raw statistics conditioned on this sysctl: for each consumer and provider { total number of operations started. total number of operations completed. time last operation completed. sum of idle-time. for each of BIO_READ, BIO_WRITE and BIO_DELETE { number of operations completed. number of bytes completed. number of ENOMEM errors. number of other errors. sum of transaction time. } } API for getting hold of these statistics data not included yet.
2003-02-07 23:08:24 +00:00
We expect 'bio_data != NULL' for BIO_{READ,WRITE,GETATTR}, but for BIO_{DELETE,FLUSH} we expect 'bio_data == NULL'. Reviewed by: phk
2007-01-28 23:36:07 +00:00
if (bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_GETATTR)) {
Add a new I/O request - BIO_FLUSH, which basically tells providers below to flush their caches. For now will mostly be used by disks to flush their write cache. Sponsored by: home.pl
2006-10-31 21:11:21 +00:00
KASSERT(bp->bio_data != NULL,
We expect 'bio_data != NULL' for BIO_{READ,WRITE,GETATTR}, but for BIO_{DELETE,FLUSH} we expect 'bio_data == NULL'. Reviewed by: phk
2007-01-28 23:36:07 +00:00
("NULL bp->data in g_io_request(cmd=%hhu)", bp->bio_cmd));
}
if (bp->bio_cmd & (BIO_DELETE|BIO_FLUSH)) {
KASSERT(bp->bio_data == NULL,
("non-NULL bp->data in g_io_request(cmd=%hhu)",
bp->bio_cmd));
Add a new I/O request - BIO_FLUSH, which basically tells providers below to flush their caches. For now will mostly be used by disks to flush their write cache. Sponsored by: home.pl
2006-10-31 21:11:21 +00:00
}
Add more KASSERTS and checks.
2004-08-30 09:33:06 +00:00
if (bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_DELETE)) {
KASSERT(bp->bio_offset % cp->provider->sectorsize == 0,
("wrong offset %jd for sectorsize %u",
bp->bio_offset, cp->provider->sectorsize));
KASSERT(bp->bio_length % cp->provider->sectorsize == 0,
("wrong length %jd for sectorsize %u",
bp->bio_length, cp->provider->sectorsize));
}
Trace information about a buffer while we still control it. Reviewed by: phk Approved by: sam (mentor)
2004-10-11 21:22:59 +00:00
g_trace(G_T_BIO, "bio_request(%p) from %p(%s) to %p(%s) cmd %d",
bp, cp, cp->geom->name, pp, pp->name, bp->bio_cmd);
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
bp->bio_from = cp;
Commit the correct copy of the g_stat structure. Add debug.sizeof.g_stat sysctl. Set the id field of the g_stat when we create consumers and providers. Remove biocount from consumer, we will use the counters in the g_stat structure instead. Replace one field which will need to be atomically manipulated with two fields which will not (stat.nop and stat.nend). Change add companion field to bio_children: bio_inbed for the exact same reason. Don't output the biocount in the confdot output. Fix KASSERT in g_io_request(). Add sysctl kern.geom.collectstats defaulting to off. Collect the following raw statistics conditioned on this sysctl: for each consumer and provider { total number of operations started. total number of operations completed. time last operation completed. sum of idle-time. for each of BIO_READ, BIO_WRITE and BIO_DELETE { number of operations completed. number of bytes completed. number of ENOMEM errors. number of other errors. sum of transaction time. } } API for getting hold of these statistics data not included yet.
2003-02-07 23:08:24 +00:00
bp->bio_to = pp;
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
bp->bio_error = 0;
bp->bio_completed = 0;
Protect the start/end counts on consumers and providers with the up/down mutexes. Make it possible to also protect the disk statistics (at a minor cost in performance) by setting bit 2 of kern.geom.collectstats.
2004-09-28 11:56:37 +00:00
KASSERT(!(bp->bio_flags & BIO_ONQUEUE),
("Bio already on queue bp=%p", bp));
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
if ((g_collectstats & G_STATS_CONSUMERS) != 0 ||
((g_collectstats & G_STATS_PROVIDERS) != 0 && pp->stat != NULL))
Do not fetch precise time of request start when stats collection disabled. Reviewed by: pjd, phk
2010-03-24 18:04:25 +00:00
binuptime(&bp->bio_t0);
else
getbinuptime(&bp->bio_t0);
By design I left a tiny race in updating the I/O statistics based on the assumption that performance was more important that beancounter quality statistics. As it transpires the microoptimization is not measurable in the real world and the inconsistent statistics confuse users, so revert the decision. MT6 candidate: possibly MT5 candidate: possibly
2005-07-25 21:12:54 +00:00
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
#ifdef GET_STACK_USAGE
direct = (cp->flags & G_CF_DIRECT_SEND) &&
(pp->flags & G_PF_DIRECT_RECEIVE) &&
!g_is_geom_thread(curthread) &&
(((pp->flags & G_PF_ACCEPT_UNMAPPED) == 0 &&
(bp->bio_flags & BIO_UNMAPPED) != 0) || THREAD_CAN_SLEEP());
if (direct) {
/* Block direct execution if less then half of stack left. */
size_t st, su;
GET_STACK_USAGE(st, su);
if (su * 2 > st)
direct = 0;
}
#else
direct = 0;
#endif
if (!TAILQ_EMPTY(&g_classifier_tailq) && !bp->bio_classifier1) {
g_bioq_lock(&g_bio_run_down);
g_run_classifiers(bp);
g_bioq_unlock(&g_bio_run_down);
}
By design I left a tiny race in updating the I/O statistics based on the assumption that performance was more important that beancounter quality statistics. As it transpires the microoptimization is not measurable in the real world and the inconsistent statistics confuse users, so revert the decision. MT6 candidate: possibly MT5 candidate: possibly
2005-07-25 21:12:54 +00:00
/*
* The statistics collection is lockless, as such, but we
* can not update one instance of the statistics from more
* than one thread at a time, so grab the lock first.
*/
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
mtxp = mtx_pool_find(mtxpool_sleep, pp);
mtx_lock(mtxp);
if (g_collectstats & G_STATS_PROVIDERS)
Protect the start/end counts on consumers and providers with the up/down mutexes. Make it possible to also protect the disk statistics (at a minor cost in performance) by setting bit 2 of kern.geom.collectstats.
2004-09-28 11:56:37 +00:00
devstat_start_transaction(pp->stat, &bp->bio_t0);
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
if (g_collectstats & G_STATS_CONSUMERS)
Protect the start/end counts on consumers and providers with the up/down mutexes. Make it possible to also protect the disk statistics (at a minor cost in performance) by setting bit 2 of kern.geom.collectstats.
2004-09-28 11:56:37 +00:00
devstat_start_transaction(cp->stat, &bp->bio_t0);
pp->nstart++;
Make the sysctl kern.geom.collectstats more granular. Bit 0 controls statistics collection on GEOM providers. Bit 1 controls statistics collection on GEOM consumers. Default value is 1. Prodded by: scottl
2004-06-09 19:44:44 +00:00
cp->nstart++;
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
mtx_unlock(mtxp);
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
if (direct) {
error = g_io_check(bp);
if (error >= 0) {
CTR3(KTR_GEOM, "g_io_request g_io_check on bp %p "
"provider %s returned %d", bp, bp->bio_to->name,
error);
g_io_deliver(bp, error);
return;
}
bp->bio_to->geom->start(bp);
} else {
g_bioq_lock(&g_bio_run_down);
first = TAILQ_EMPTY(&g_bio_run_down.bio_queue);
TAILQ_INSERT_TAIL(&g_bio_run_down.bio_queue, bp, bio_queue);
bp->bio_flags |= BIO_ONQUEUE;
g_bio_run_down.bio_queue_length++;
g_bioq_unlock(&g_bio_run_down);
/* Pass it on down. */
if (first)
wakeup(&g_wait_down);
}
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
}
void
Retire g_io_fail() and let g_io_deliver() take an error argument instead. Sponsored by: DARPA & NAI Labs.
2002-09-30 08:54:46 +00:00
g_io_deliver(struct bio *bp, int error)
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
{
MFprojects/camlock r254905: Introduce new function devstat_end_transaction_bio_bt(), adding new argument to specify present time. Use this function to move binuptime() out of lock, substantially reducing lock congestion when slow timecounter is used.
2013-10-16 09:12:40 +00:00
struct bintime now;
Commit the correct copy of the g_stat structure. Add debug.sizeof.g_stat sysctl. Set the id field of the g_stat when we create consumers and providers. Remove biocount from consumer, we will use the counters in the g_stat structure instead. Replace one field which will need to be atomically manipulated with two fields which will not (stat.nop and stat.nend). Change add companion field to bio_children: bio_inbed for the exact same reason. Don't output the biocount in the confdot output. Fix KASSERT in g_io_request(). Add sysctl kern.geom.collectstats defaulting to off. Collect the following raw statistics conditioned on this sysctl: for each consumer and provider { total number of operations started. total number of operations completed. time last operation completed. sum of idle-time. for each of BIO_READ, BIO_WRITE and BIO_DELETE { number of operations completed. number of bytes completed. number of ENOMEM errors. number of other errors. sum of transaction time. } } API for getting hold of these statistics data not included yet.
2003-02-07 23:08:24 +00:00
struct g_consumer *cp;
struct g_provider *pp;
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
struct mtx *mtxp;
int direct, first;
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
Reorder a couple of KASSERTS to give more sensible messages. Found by: GEOM 101 class of '03
2003-09-11 00:49:02 +00:00
KASSERT(bp != NULL, ("NULL bp in g_io_deliver"));
Commit the correct copy of the g_stat structure. Add debug.sizeof.g_stat sysctl. Set the id field of the g_stat when we create consumers and providers. Remove biocount from consumer, we will use the counters in the g_stat structure instead. Replace one field which will need to be atomically manipulated with two fields which will not (stat.nop and stat.nend). Change add companion field to bio_children: bio_inbed for the exact same reason. Don't output the biocount in the confdot output. Fix KASSERT in g_io_request(). Add sysctl kern.geom.collectstats defaulting to off. Collect the following raw statistics conditioned on this sysctl: for each consumer and provider { total number of operations started. total number of operations completed. time last operation completed. sum of idle-time. for each of BIO_READ, BIO_WRITE and BIO_DELETE { number of operations completed. number of bytes completed. number of ENOMEM errors. number of other errors. sum of transaction time. } } API for getting hold of these statistics data not included yet.
2003-02-07 23:08:24 +00:00
pp = bp->bio_to;
Allow our bio tools to be used for local bio-chopping by not mandating a bio_from value. bio_to is still mandated (mostly for debuggign) and shall be copied from the parent bio.
2003-10-06 09:07:35 +00:00
KASSERT(pp != NULL, ("NULL bio_to in g_io_deliver"));
cp = bp->bio_from;
if (cp == NULL) {
bp->bio_error = error;
bp->bio_done(bp);
return;
}
Commit the correct copy of the g_stat structure. Add debug.sizeof.g_stat sysctl. Set the id field of the g_stat when we create consumers and providers. Remove biocount from consumer, we will use the counters in the g_stat structure instead. Replace one field which will need to be atomically manipulated with two fields which will not (stat.nop and stat.nend). Change add companion field to bio_children: bio_inbed for the exact same reason. Don't output the biocount in the confdot output. Fix KASSERT in g_io_request(). Add sysctl kern.geom.collectstats defaulting to off. Collect the following raw statistics conditioned on this sysctl: for each consumer and provider { total number of operations started. total number of operations completed. time last operation completed. sum of idle-time. for each of BIO_READ, BIO_WRITE and BIO_DELETE { number of operations completed. number of bytes completed. number of ENOMEM errors. number of other errors. sum of transaction time. } } API for getting hold of these statistics data not included yet.
2003-02-07 23:08:24 +00:00
KASSERT(cp != NULL, ("NULL bio_from in g_io_deliver"));
KASSERT(cp->geom != NULL, ("NULL bio_from->geom in g_io_deliver"));
Make gjournal work with kernel compiled with "options DIAGNOSTIC". Previously, it would panic immediately. Reviewed by: pjd Approved by: re (kib)
2009-06-30 14:34:06 +00:00
#ifdef DIAGNOSTIC
/*
* Some classes - GJournal in particular - can modify bio's
* private fields while the bio is in transit; G_GEOM_VOLATILE_BIO
* flag means it's an expected behaviour for that particular geom.
*/
if ((cp->geom->flags & G_GEOM_VOLATILE_BIO) == 0) {
KASSERT(bp->bio_caller1 == bp->_bio_caller1,
("bio_caller1 used by the provider %s", pp->name));
KASSERT(bp->bio_caller2 == bp->_bio_caller2,
("bio_caller2 used by the provider %s", pp->name));
KASSERT(bp->bio_cflags == bp->_bio_cflags,
("bio_cflags used by the provider %s", pp->name));
}
#endif
Calculate bio_completed properly or die! Approved by: phk
2004-04-04 20:37:28 +00:00
KASSERT(bp->bio_completed >= 0, ("bio_completed can't be less than 0"));
KASSERT(bp->bio_completed <= bp->bio_length,
("bio_completed can't be greater than bio_length"));
white-space changes
2002-12-26 21:02:50 +00:00
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
g_trace(G_T_BIO,
white-space changes
2002-12-26 21:02:50 +00:00
"g_io_deliver(%p) from %p(%s) to %p(%s) cmd %d error %d off %jd len %jd",
Commit the correct copy of the g_stat structure. Add debug.sizeof.g_stat sysctl. Set the id field of the g_stat when we create consumers and providers. Remove biocount from consumer, we will use the counters in the g_stat structure instead. Replace one field which will need to be atomically manipulated with two fields which will not (stat.nop and stat.nend). Change add companion field to bio_children: bio_inbed for the exact same reason. Don't output the biocount in the confdot output. Fix KASSERT in g_io_request(). Add sysctl kern.geom.collectstats defaulting to off. Collect the following raw statistics conditioned on this sysctl: for each consumer and provider { total number of operations started. total number of operations completed. time last operation completed. sum of idle-time. for each of BIO_READ, BIO_WRITE and BIO_DELETE { number of operations completed. number of bytes completed. number of ENOMEM errors. number of other errors. sum of transaction time. } } API for getting hold of these statistics data not included yet.
2003-02-07 23:08:24 +00:00
bp, cp, cp->geom->name, pp, pp->name, bp->bio_cmd, error,
Don't track bio allocation in debug output. Sponsored by: DARPA & NAI Labs.
2002-10-20 08:45:17 +00:00
(intmax_t)bp->bio_offset, (intmax_t)bp->bio_length);
Commit the correct copy of the g_stat structure. Add debug.sizeof.g_stat sysctl. Set the id field of the g_stat when we create consumers and providers. Remove biocount from consumer, we will use the counters in the g_stat structure instead. Replace one field which will need to be atomically manipulated with two fields which will not (stat.nop and stat.nend). Change add companion field to bio_children: bio_inbed for the exact same reason. Don't output the biocount in the confdot output. Fix KASSERT in g_io_request(). Add sysctl kern.geom.collectstats defaulting to off. Collect the following raw statistics conditioned on this sysctl: for each consumer and provider { total number of operations started. total number of operations completed. time last operation completed. sum of idle-time. for each of BIO_READ, BIO_WRITE and BIO_DELETE { number of operations completed. number of bytes completed. number of ENOMEM errors. number of other errors. sum of transaction time. } } API for getting hold of these statistics data not included yet.
2003-02-07 23:08:24 +00:00
Protect the start/end counts on consumers and providers with the up/down mutexes. Make it possible to also protect the disk statistics (at a minor cost in performance) by setting bit 2 of kern.geom.collectstats.
2004-09-28 11:56:37 +00:00
KASSERT(!(bp->bio_flags & BIO_ONQUEUE),
("Bio already on queue bp=%p", bp));
Add more KASSERTS and checks.
2004-08-30 09:33:06 +00:00
/*
* XXX: next two doesn't belong here
*/
Retire the GEOM private statistics code and use devstat instead.
2003-03-18 09:42:33 +00:00
bp->bio_bcount = bp->bio_length;
Make the sysctl kern.geom.collectstats more granular. Bit 0 controls statistics collection on GEOM providers. Bit 1 controls statistics collection on GEOM consumers. Default value is 1. Prodded by: scottl
2004-06-09 19:44:44 +00:00
bp->bio_resid = bp->bio_bcount - bp->bio_completed;
Protect the start/end counts on consumers and providers with the up/down mutexes. Make it possible to also protect the disk statistics (at a minor cost in performance) by setting bit 2 of kern.geom.collectstats.
2004-09-28 11:56:37 +00:00
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
#ifdef GET_STACK_USAGE
direct = (pp->flags & G_PF_DIRECT_SEND) &&
(cp->flags & G_CF_DIRECT_RECEIVE) &&
!g_is_geom_thread(curthread);
if (direct) {
/* Block direct execution if less then half of stack left. */
size_t st, su;
GET_STACK_USAGE(st, su);
if (su * 2 > st)
direct = 0;
}
#else
direct = 0;
#endif
By design I left a tiny race in updating the I/O statistics based on the assumption that performance was more important that beancounter quality statistics. As it transpires the microoptimization is not measurable in the real world and the inconsistent statistics confuse users, so revert the decision. MT6 candidate: possibly MT5 candidate: possibly
2005-07-25 21:12:54 +00:00
/*
* The statistics collection is lockless, as such, but we
* can not update one instance of the statistics from more
* than one thread at a time, so grab the lock first.
*/
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
if ((g_collectstats & G_STATS_CONSUMERS) != 0 ||
((g_collectstats & G_STATS_PROVIDERS) != 0 && pp->stat != NULL))
MFprojects/camlock r254905: Introduce new function devstat_end_transaction_bio_bt(), adding new argument to specify present time. Use this function to move binuptime() out of lock, substantially reducing lock congestion when slow timecounter is used.
2013-10-16 09:12:40 +00:00
binuptime(&now);
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
mtxp = mtx_pool_find(mtxpool_sleep, cp);
mtx_lock(mtxp);
if (g_collectstats & G_STATS_PROVIDERS)
MFprojects/camlock r254905: Introduce new function devstat_end_transaction_bio_bt(), adding new argument to specify present time. Use this function to move binuptime() out of lock, substantially reducing lock congestion when slow timecounter is used.
2013-10-16 09:12:40 +00:00
devstat_end_transaction_bio_bt(pp->stat, bp, &now);
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
if (g_collectstats & G_STATS_CONSUMERS)
MFprojects/camlock r254905: Introduce new function devstat_end_transaction_bio_bt(), adding new argument to specify present time. Use this function to move binuptime() out of lock, substantially reducing lock congestion when slow timecounter is used.
2013-10-16 09:12:40 +00:00
devstat_end_transaction_bio_bt(cp->stat, bp, &now);
Don't abuse the statistics counters for detecting if we have outstanding I/O requests, instead use the new dedicated fields in the consumer and provider to track this.
2003-03-09 09:59:48 +00:00
cp->nend++;
pp->nend++;
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
mtx_unlock(mtxp);
Protect the start/end counts on consumers and providers with the up/down mutexes. Make it possible to also protect the disk statistics (at a minor cost in performance) by setting bit 2 of kern.geom.collectstats.
2004-09-28 11:56:37 +00:00
if (error != ENOMEM) {
bp->bio_error = error;
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
if (direct) {
biodone(bp);
} else {
g_bioq_lock(&g_bio_run_up);
first = TAILQ_EMPTY(&g_bio_run_up.bio_queue);
TAILQ_INSERT_TAIL(&g_bio_run_up.bio_queue, bp, bio_queue);
bp->bio_flags |= BIO_ONQUEUE;
g_bio_run_up.bio_queue_length++;
g_bioq_unlock(&g_bio_run_up);
if (first)
wakeup(&g_wait_up);
}
malloc(9) with M_NOWAIT seems to return NULL a lot more than I would have expected under -current. This is a problem for GEOM because the up/down threads cannot sleep waiting for memory to become free. The reason they cannot sleep is that paging things out to disk may be the only way we can clear up some RAM. Nice catch-22 there. Implement a rudimentary ENOMEM recovery strategy: If an I/O request fails with an error code of ENOMEM, schedule it for a retry, and tell the down-thread to sleep hz/10 to get other parts of the system a chance to free up some memory, in particular the up-path in GEOM. All caches should probably start to monitor malloc(9) failures using the new malloc_last_fail() function, and release when it indicates congestion. Sponsored by: DARPA & NAI Labs.
2002-11-02 11:08:07 +00:00
return;
}
Protect the start/end counts on consumers and providers with the up/down mutexes. Make it possible to also protect the disk statistics (at a minor cost in performance) by setting bit 2 of kern.geom.collectstats.
2004-09-28 11:56:37 +00:00
if (bootverbose)
printf("ENOMEM %p on %p(%s)\n", bp, pp, pp->name);
bp->bio_children = 0;
bp->bio_inbed = 0;
Reset provider-specific fields when resending I/O request in low memory conditions. This fixes assertion which checks those fields when kernel is compiled with DIAGNOSTIC. Reported by: kib, pho MFC after: 1 week
2012-12-26 20:07:47 +00:00
bp->bio_driver1 = NULL;
bp->bio_driver2 = NULL;
bp->bio_pflags = 0;
Protect the start/end counts on consumers and providers with the up/down mutexes. Make it possible to also protect the disk statistics (at a minor cost in performance) by setting bit 2 of kern.geom.collectstats.
2004-09-28 11:56:37 +00:00
g_io_request(bp, cp);
pace++;
return;
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
}
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
SYSCTL_DECL(_kern_geom);
static long transient_maps;
SYSCTL_LONG(_kern_geom, OID_AUTO, transient_maps, CTLFLAG_RD,
&transient_maps, 0,
"Total count of the transient mapping requests");
u_int transient_map_retries = 10;
SYSCTL_UINT(_kern_geom, OID_AUTO, transient_map_retries, CTLFLAG_RW,
&transient_map_retries, 0,
"Max count of retries used before giving up on creating transient map");
int transient_map_hard_failures;
SYSCTL_INT(_kern_geom, OID_AUTO, transient_map_hard_failures, CTLFLAG_RD,
&transient_map_hard_failures, 0,
"Failures to establish the transient mapping due to retry attempts "
"exhausted");
int transient_map_soft_failures;
SYSCTL_INT(_kern_geom, OID_AUTO, transient_map_soft_failures, CTLFLAG_RD,
&transient_map_soft_failures, 0,
"Count of retried failures to establish the transient mapping");
int inflight_transient_maps;
SYSCTL_INT(_kern_geom, OID_AUTO, inflight_transient_maps, CTLFLAG_RD,
&inflight_transient_maps, 0,
"Current count of the active transient maps");
static int
g_io_transient_map_bio(struct bio *bp)
{
vm_offset_t addr;
long size;
u_int retried;
Assert that transient mapping of the bio is only done when unmapped buffers are allowed. Sponsored by: The FreeBSD Foundation
2013-03-21 07:26:33 +00:00
KASSERT(unmapped_buf_allowed, ("unmapped disabled"));
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
size = round_page(bp->bio_ma_offset + bp->bio_length);
KASSERT(size / PAGE_SIZE == bp->bio_ma_n, ("Bio too short %p", bp));
addr = 0;
retried = 0;
atomic_add_long(&transient_maps, 1);
retry:
- Add a general purpose resource allocator, vmem, from NetBSD. It was originally inspired by the Solaris vmem detailed in the proceedings of usenix 2001. The NetBSD version was heavily refactored for bugs and simplicity. - Use this resource allocator to allocate the buffer and transient maps. Buffer cache defrags are reduced by 25% when used by filesystems with mixed block sizes. Ultimately this may permit dynamic buffer cache sizing on low KVA machines. Discussed with: alc, kib, attilio Tested by: pho Sponsored by: EMC / Isilon Storage Division
2013-06-28 03:51:20 +00:00
if (vmem_alloc(transient_arena, size, M_BESTFIT | M_NOWAIT, &addr)) {
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
if (transient_map_retries != 0 &&
retried >= transient_map_retries) {
CTR2(KTR_GEOM, "g_down cannot map bp %p provider %s",
bp, bp->bio_to->name);
atomic_add_int(&transient_map_hard_failures, 1);
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
return (EDEADLK/* XXXKIB */);
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
} else {
/*
* Naive attempt to quisce the I/O to get more
* in-flight requests completed and defragment
- Add a general purpose resource allocator, vmem, from NetBSD. It was originally inspired by the Solaris vmem detailed in the proceedings of usenix 2001. The NetBSD version was heavily refactored for bugs and simplicity. - Use this resource allocator to allocate the buffer and transient maps. Buffer cache defrags are reduced by 25% when used by filesystems with mixed block sizes. Ultimately this may permit dynamic buffer cache sizing on low KVA machines. Discussed with: alc, kib, attilio Tested by: pho Sponsored by: EMC / Isilon Storage Division
2013-06-28 03:51:20 +00:00
* the transient_arena.
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
*/
CTR3(KTR_GEOM, "g_down retrymap bp %p provider %s r %d",
bp, bp->bio_to->name, retried);
pause("g_d_tra", hz / 10);
retried++;
atomic_add_int(&transient_map_soft_failures, 1);
goto retry;
}
}
atomic_add_int(&inflight_transient_maps, 1);
pmap_qenter((vm_offset_t)addr, bp->bio_ma, OFF_TO_IDX(size));
bp->bio_data = (caddr_t)addr + bp->bio_ma_offset;
bp->bio_flags |= BIO_TRANSIENT_MAPPING;
bp->bio_flags &= ~BIO_UNMAPPED;
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
return (EJUSTRETURN);
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
}
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
void
g_io_schedule_down(struct thread *tp __unused)
{
struct bio *bp;
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
int error;
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
for(;;) {
Implement a bio-taskqueue to reduce number of context switches in disk I/O processing. The intent is that the disk driver in its hardware interrupt routine will simply schedule the bio on the task queue with a routine to finish off whatever needs done. The g_up thread will then schedule this routine, the likely outcome of which is a biodone() which queues the bio on g_up's regular queue where it will be picked up and processed. Compared to the using the regular taskqueue, this saves one contextswitch. Change our scheduling of the g_up and g_down queues to be water-tight, at the cost of breaking the userland regression test-shims. Input and ideas from: scottl
2003-02-11 22:30:26 +00:00
g_bioq_lock(&g_bio_run_down);
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
bp = g_bioq_first(&g_bio_run_down);
Implement a bio-taskqueue to reduce number of context switches in disk I/O processing. The intent is that the disk driver in its hardware interrupt routine will simply schedule the bio on the task queue with a routine to finish off whatever needs done. The g_up thread will then schedule this routine, the likely outcome of which is a biodone() which queues the bio on g_up's regular queue where it will be picked up and processed. Compared to the using the regular taskqueue, this saves one contextswitch. Change our scheduling of the g_up and g_down queues to be water-tight, at the cost of breaking the userland regression test-shims. Input and ideas from: scottl
2003-02-11 22:30:26 +00:00
if (bp == NULL) {
Add KTR_GEOM, which allows tracing of basic GEOM I/O events occuring in the g_up and g_down threads. Each time a bio is propelled up and down the stack, an event is generating showing the provider, offset, and length, as well as thread wakeup and work status information.
2004-10-21 18:35:24 +00:00
CTR0(KTR_GEOM, "g_down going to sleep");
Implement a bio-taskqueue to reduce number of context switches in disk I/O processing. The intent is that the disk driver in its hardware interrupt routine will simply schedule the bio on the task queue with a routine to finish off whatever needs done. The g_up thread will then schedule this routine, the likely outcome of which is a biodone() which queues the bio on g_up's regular queue where it will be picked up and processed. Compared to the using the regular taskqueue, this saves one contextswitch. Change our scheduling of the g_up and g_down queues to be water-tight, at the cost of breaking the userland regression test-shims. Input and ideas from: scottl
2003-02-11 22:30:26 +00:00
msleep(&g_wait_down, &g_bio_run_down.bio_queue_lock,
MFp4: Remove msleep() timeout from g_io_schedule_up/down(). It works fine without it, saving few percents of CPU on high request rates without need to rearm callout twice per request.
2009-09-06 19:33:13 +00:00
PRIBIO | PDROP, "-", 0);
Implement a bio-taskqueue to reduce number of context switches in disk I/O processing. The intent is that the disk driver in its hardware interrupt routine will simply schedule the bio on the task queue with a routine to finish off whatever needs done. The g_up thread will then schedule this routine, the likely outcome of which is a biodone() which queues the bio on g_up's regular queue where it will be picked up and processed. Compared to the using the regular taskqueue, this saves one contextswitch. Change our scheduling of the g_up and g_down queues to be water-tight, at the cost of breaking the userland regression test-shims. Input and ideas from: scottl
2003-02-11 22:30:26 +00:00
continue;
}
Add KTR_GEOM, which allows tracing of basic GEOM I/O events occuring in the g_up and g_down threads. Each time a bio is propelled up and down the stack, an event is generating showing the provider, offset, and length, as well as thread wakeup and work status information.
2004-10-21 18:35:24 +00:00
CTR0(KTR_GEOM, "g_down has work to do");
Implement a bio-taskqueue to reduce number of context switches in disk I/O processing. The intent is that the disk driver in its hardware interrupt routine will simply schedule the bio on the task queue with a routine to finish off whatever needs done. The g_up thread will then schedule this routine, the likely outcome of which is a biodone() which queues the bio on g_up's regular queue where it will be picked up and processed. Compared to the using the regular taskqueue, this saves one contextswitch. Change our scheduling of the g_up and g_down queues to be water-tight, at the cost of breaking the userland regression test-shims. Input and ideas from: scottl
2003-02-11 22:30:26 +00:00
g_bioq_unlock(&g_bio_run_down);
Fix a bug in the ENOMEM pacing code which probably made it panic systems after a lot of ENOMEM errors.
2003-03-29 22:34:37 +00:00
if (pace > 0) {
Add KTR_GEOM, which allows tracing of basic GEOM I/O events occuring in the g_up and g_down threads. Each time a bio is propelled up and down the stack, an event is generating showing the provider, offset, and length, as well as thread wakeup and work status information.
2004-10-21 18:35:24 +00:00
CTR1(KTR_GEOM, "g_down pacing self (pace %d)", pace);
Use pause() rather than tsleep() on stack variables and function pointers.
2007-02-27 17:23:29 +00:00
pause("g_down", hz/10);
Fix a bug in the ENOMEM pacing code which probably made it panic systems after a lot of ENOMEM errors.
2003-03-29 22:34:37 +00:00
pace--;
}
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
CTR2(KTR_GEOM, "g_down processing bp %p provider %s", bp,
bp->bio_to->name);
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
error = g_io_check(bp);
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
if (error >= 0) {
Add KTR_GEOM, which allows tracing of basic GEOM I/O events occuring in the g_up and g_down threads. Each time a bio is propelled up and down the stack, an event is generating showing the provider, offset, and length, as well as thread wakeup and work status information.
2004-10-21 18:35:24 +00:00
CTR3(KTR_GEOM, "g_down g_io_check on bp %p provider "
"%s returned %d", bp, bp->bio_to->name, error);
Put the checks we perform on a bio before calling ::start in their own function, handle all validation and truncation at the time we process the bio instead of when it gets scheduled.
2003-02-06 21:01:36 +00:00
g_io_deliver(bp, error);
continue;
}
- Add a new simple facility for marking the current thread as being in a state where sleeping on a sleep queue is not allowed. The facility doesn't support recursion but uses a simple private per-thread flag (TDP_NOSLEEPING). The sleepq_add() function will panic if the flag is set and INVARIANTS is enabled. - Use this new facility to replace the g_xup and g_xdown mutexes that were (ab)used to achieve similar behavior. - Disallow sleeping in interrupt threads when invoking interrupt handlers. MFC after: 1 week Reviewed by: phk
2005-09-15 19:05:37 +00:00
THREAD_NO_SLEEPING();
Add KTR_GEOM, which allows tracing of basic GEOM I/O events occuring in the g_up and g_down threads. Each time a bio is propelled up and down the stack, an event is generating showing the provider, offset, and length, as well as thread wakeup and work status information.
2004-10-21 18:35:24 +00:00
CTR4(KTR_GEOM, "g_down starting bp %p provider %s off %ld "
"len %ld", bp, bp->bio_to->name, bp->bio_offset,
bp->bio_length);
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
bp->bio_to->geom->start(bp);
- Add a new simple facility for marking the current thread as being in a state where sleeping on a sleep queue is not allowed. The facility doesn't support recursion but uses a simple private per-thread flag (TDP_NOSLEEPING). The sleepq_add() function will panic if the flag is set and INVARIANTS is enabled. - Use this new facility to replace the g_xup and g_xdown mutexes that were (ab)used to achieve similar behavior. - Disallow sleeping in interrupt threads when invoking interrupt handlers. MFC after: 1 week Reviewed by: phk
2005-09-15 19:05:37 +00:00
THREAD_SLEEPING_OK();
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
}
}
Bring back the geom_bioqueues, they _are_ a good idea. ATA will uses these RSN.
2004-01-28 08:39:18 +00:00
void
bio_taskqueue(struct bio *bp, bio_task_t *func, void *arg)
{
bp->bio_task = func;
bp->bio_task_arg = arg;
/*
* The taskqueue is actually just a second queue off the "up"
* queue, so we use the same lock.
*/
g_bioq_lock(&g_bio_run_up);
Add more KASSERTS and checks.
2004-08-30 09:33:06 +00:00
KASSERT(!(bp->bio_flags & BIO_ONQUEUE),
("Bio already on queue bp=%p target taskq", bp));
bp->bio_flags |= BIO_ONQUEUE;
Bring back the geom_bioqueues, they _are_ a good idea. ATA will uses these RSN.
2004-01-28 08:39:18 +00:00
TAILQ_INSERT_TAIL(&g_bio_run_task.bio_queue, bp, bio_queue);
g_bio_run_task.bio_queue_length++;
wakeup(&g_wait_up);
g_bioq_unlock(&g_bio_run_up);
}
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
void
g_io_schedule_up(struct thread *tp __unused)
{
struct bio *bp;
for(;;) {
Implement a bio-taskqueue to reduce number of context switches in disk I/O processing. The intent is that the disk driver in its hardware interrupt routine will simply schedule the bio on the task queue with a routine to finish off whatever needs done. The g_up thread will then schedule this routine, the likely outcome of which is a biodone() which queues the bio on g_up's regular queue where it will be picked up and processed. Compared to the using the regular taskqueue, this saves one contextswitch. Change our scheduling of the g_up and g_down queues to be water-tight, at the cost of breaking the userland regression test-shims. Input and ideas from: scottl
2003-02-11 22:30:26 +00:00
g_bioq_lock(&g_bio_run_up);
Bring back the geom_bioqueues, they _are_ a good idea. ATA will uses these RSN.
2004-01-28 08:39:18 +00:00
bp = g_bioq_first(&g_bio_run_task);
if (bp != NULL) {
g_bioq_unlock(&g_bio_run_up);
- Add a new simple facility for marking the current thread as being in a state where sleeping on a sleep queue is not allowed. The facility doesn't support recursion but uses a simple private per-thread flag (TDP_NOSLEEPING). The sleepq_add() function will panic if the flag is set and INVARIANTS is enabled. - Use this new facility to replace the g_xup and g_xdown mutexes that were (ab)used to achieve similar behavior. - Disallow sleeping in interrupt threads when invoking interrupt handlers. MFC after: 1 week Reviewed by: phk
2005-09-15 19:05:37 +00:00
THREAD_NO_SLEEPING();
Add KTR_GEOM, which allows tracing of basic GEOM I/O events occuring in the g_up and g_down threads. Each time a bio is propelled up and down the stack, an event is generating showing the provider, offset, and length, as well as thread wakeup and work status information.
2004-10-21 18:35:24 +00:00
CTR1(KTR_GEOM, "g_up processing task bp %p", bp);
Bring back the geom_bioqueues, they _are_ a good idea. ATA will uses these RSN.
2004-01-28 08:39:18 +00:00
bp->bio_task(bp->bio_task_arg);
- Add a new simple facility for marking the current thread as being in a state where sleeping on a sleep queue is not allowed. The facility doesn't support recursion but uses a simple private per-thread flag (TDP_NOSLEEPING). The sleepq_add() function will panic if the flag is set and INVARIANTS is enabled. - Use this new facility to replace the g_xup and g_xdown mutexes that were (ab)used to achieve similar behavior. - Disallow sleeping in interrupt threads when invoking interrupt handlers. MFC after: 1 week Reviewed by: phk
2005-09-15 19:05:37 +00:00
THREAD_SLEEPING_OK();
Bring back the geom_bioqueues, they _are_ a good idea. ATA will uses these RSN.
2004-01-28 08:39:18 +00:00
continue;
}
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
bp = g_bioq_first(&g_bio_run_up);
Implement a bio-taskqueue to reduce number of context switches in disk I/O processing. The intent is that the disk driver in its hardware interrupt routine will simply schedule the bio on the task queue with a routine to finish off whatever needs done. The g_up thread will then schedule this routine, the likely outcome of which is a biodone() which queues the bio on g_up's regular queue where it will be picked up and processed. Compared to the using the regular taskqueue, this saves one contextswitch. Change our scheduling of the g_up and g_down queues to be water-tight, at the cost of breaking the userland regression test-shims. Input and ideas from: scottl
2003-02-11 22:30:26 +00:00
if (bp != NULL) {
g_bioq_unlock(&g_bio_run_up);
- Add a new simple facility for marking the current thread as being in a state where sleeping on a sleep queue is not allowed. The facility doesn't support recursion but uses a simple private per-thread flag (TDP_NOSLEEPING). The sleepq_add() function will panic if the flag is set and INVARIANTS is enabled. - Use this new facility to replace the g_xup and g_xdown mutexes that were (ab)used to achieve similar behavior. - Disallow sleeping in interrupt threads when invoking interrupt handlers. MFC after: 1 week Reviewed by: phk
2005-09-15 19:05:37 +00:00
THREAD_NO_SLEEPING();
Add KTR_GEOM, which allows tracing of basic GEOM I/O events occuring in the g_up and g_down threads. Each time a bio is propelled up and down the stack, an event is generating showing the provider, offset, and length, as well as thread wakeup and work status information.
2004-10-21 18:35:24 +00:00
CTR4(KTR_GEOM, "g_up biodone bp %p provider %s off "
Just a fixup for a KTRACE message I stumbled upon many moons ago. Reviewed by: Scott Long MFC after: 2 days
2008-09-18 15:02:19 +00:00
"%jd len %ld", bp, bp->bio_to->name,
Add KTR_GEOM, which allows tracing of basic GEOM I/O events occuring in the g_up and g_down threads. Each time a bio is propelled up and down the stack, an event is generating showing the provider, offset, and length, as well as thread wakeup and work status information.
2004-10-21 18:35:24 +00:00
bp->bio_offset, bp->bio_length);
Implement a bio-taskqueue to reduce number of context switches in disk I/O processing. The intent is that the disk driver in its hardware interrupt routine will simply schedule the bio on the task queue with a routine to finish off whatever needs done. The g_up thread will then schedule this routine, the likely outcome of which is a biodone() which queues the bio on g_up's regular queue where it will be picked up and processed. Compared to the using the regular taskqueue, this saves one contextswitch. Change our scheduling of the g_up and g_down queues to be water-tight, at the cost of breaking the userland regression test-shims. Input and ideas from: scottl
2003-02-11 22:30:26 +00:00
biodone(bp);
- Add a new simple facility for marking the current thread as being in a state where sleeping on a sleep queue is not allowed. The facility doesn't support recursion but uses a simple private per-thread flag (TDP_NOSLEEPING). The sleepq_add() function will panic if the flag is set and INVARIANTS is enabled. - Use this new facility to replace the g_xup and g_xdown mutexes that were (ab)used to achieve similar behavior. - Disallow sleeping in interrupt threads when invoking interrupt handlers. MFC after: 1 week Reviewed by: phk
2005-09-15 19:05:37 +00:00
THREAD_SLEEPING_OK();
Implement a bio-taskqueue to reduce number of context switches in disk I/O processing. The intent is that the disk driver in its hardware interrupt routine will simply schedule the bio on the task queue with a routine to finish off whatever needs done. The g_up thread will then schedule this routine, the likely outcome of which is a biodone() which queues the bio on g_up's regular queue where it will be picked up and processed. Compared to the using the regular taskqueue, this saves one contextswitch. Change our scheduling of the g_up and g_down queues to be water-tight, at the cost of breaking the userland regression test-shims. Input and ideas from: scottl
2003-02-11 22:30:26 +00:00
continue;
}
Add KTR_GEOM, which allows tracing of basic GEOM I/O events occuring in the g_up and g_down threads. Each time a bio is propelled up and down the stack, an event is generating showing the provider, offset, and length, as well as thread wakeup and work status information.
2004-10-21 18:35:24 +00:00
CTR0(KTR_GEOM, "g_up going to sleep");
Implement a bio-taskqueue to reduce number of context switches in disk I/O processing. The intent is that the disk driver in its hardware interrupt routine will simply schedule the bio on the task queue with a routine to finish off whatever needs done. The g_up thread will then schedule this routine, the likely outcome of which is a biodone() which queues the bio on g_up's regular queue where it will be picked up and processed. Compared to the using the regular taskqueue, this saves one contextswitch. Change our scheduling of the g_up and g_down queues to be water-tight, at the cost of breaking the userland regression test-shims. Input and ideas from: scottl
2003-02-11 22:30:26 +00:00
msleep(&g_wait_up, &g_bio_run_up.bio_queue_lock,
MFp4: Remove msleep() timeout from g_io_schedule_up/down(). It works fine without it, saving few percents of CPU on high request rates without need to rearm callout twice per request.
2009-09-06 19:33:13 +00:00
PRIBIO | PDROP, "-", 0);
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
}
}
void *
g_read_data(struct g_consumer *cp, off_t offset, off_t length, int *error)
{
struct bio *bp;
void *ptr;
int errorc;
- Set maximum request size to MAXPHYS (128kB), instead of DFLPHYS (64kB). - Set minimum request size to sectorsize, instead of 512 bytes. Approved by: phk (some time ago)
2004-09-28 08:34:27 +00:00
KASSERT(length > 0 && length >= cp->provider->sectorsize &&
length <= MAXPHYS, ("g_read_data(): invalid length %jd",
(intmax_t)length));
Add more KASSERTS().
2003-09-26 20:52:46 +00:00
Introduce g_alloc_bio() as a waiting variant of g_new_bio(). Use in places where we can sleep and where we previously failed to check for a NULL pointer. MT5 candidate.
2004-08-27 14:43:11 +00:00
bp = g_alloc_bio();
For now, don't wait for drives to stop returning EBUSY. There is too much broken harware around it seems. Sponsored by: DARPA & NAI Labs.
2002-10-08 07:03:58 +00:00
bp->bio_cmd = BIO_READ;
bp->bio_done = NULL;
bp->bio_offset = offset;
bp->bio_length = length;
Back out M_* changes, per decision of the TRB. Approved by: trb
2003-02-19 05:47:46 +00:00
ptr = g_malloc(length, M_WAITOK);
For now, don't wait for drives to stop returning EBUSY. There is too much broken harware around it seems. Sponsored by: DARPA & NAI Labs.
2002-10-08 07:03:58 +00:00
bp->bio_data = ptr;
g_io_request(bp, cp);
errorc = biowait(bp, "gread");
if (error != NULL)
*error = errorc;
g_destroy_bio(bp);
if (errorc) {
g_free(ptr);
ptr = NULL;
}
First commit of the GEOM subsystem to make it easier for people to test and play with this. This is not yet production quality and should be run only on dedicated test boxes. For people who want to develop transformations for GEOM there exist a set of shims to run geom in userland (ask phk@freebsd.org). Reports of all kinds to: phk@freebsd.org Please include in report: dmesg sysctl debug.geomdot sysctl debug.geomconf Known significant limitations: no kernel dump facility. ioctls severely restricted. Sponsored by: DARPA, NAI Labs
2002-03-11 21:42:35 +00:00
return (ptr);
}
Introduce g_write_data() function. Sponsored by: DARPA & NAI Labs
2002-09-30 08:50:47 +00:00
int
g_write_data(struct g_consumer *cp, off_t offset, void *ptr, off_t length)
{
struct bio *bp;
int error;
- Set maximum request size to MAXPHYS (128kB), instead of DFLPHYS (64kB). - Set minimum request size to sectorsize, instead of 512 bytes. Approved by: phk (some time ago)
2004-09-28 08:34:27 +00:00
KASSERT(length > 0 && length >= cp->provider->sectorsize &&
length <= MAXPHYS, ("g_write_data(): invalid length %jd",
(intmax_t)length));
Add more KASSERTS().
2003-09-26 20:52:46 +00:00
Introduce g_alloc_bio() as a waiting variant of g_new_bio(). Use in places where we can sleep and where we previously failed to check for a NULL pointer. MT5 candidate.
2004-08-27 14:43:11 +00:00
bp = g_alloc_bio();
Introduce g_write_data() function. Sponsored by: DARPA & NAI Labs
2002-09-30 08:50:47 +00:00
bp->bio_cmd = BIO_WRITE;
bp->bio_done = NULL;
bp->bio_offset = offset;
bp->bio_length = length;
bp->bio_data = ptr;
g_io_request(bp, cp);
error = biowait(bp, "gwrite");
g_destroy_bio(bp);
return (error);
}
Added g_print_bio() function to print informations about given bio. Approved by: phk, scottl (mentor)
2004-02-11 18:21:32 +00:00
Implement g_delete_data() similar to g_read_data() and g_write_data(). OK'ed by: phk
2007-05-05 16:35:22 +00:00
int
g_delete_data(struct g_consumer *cp, off_t offset, off_t length)
{
struct bio *bp;
int error;
Don't limit BIO_DELETE requests to MAXPHYS, they perform no data transfers, so they are not subject to the VM system limitation.
2007-12-16 18:03:31 +00:00
KASSERT(length > 0 && length >= cp->provider->sectorsize,
("g_delete_data(): invalid length %jd", (intmax_t)length));
Implement g_delete_data() similar to g_read_data() and g_write_data(). OK'ed by: phk
2007-05-05 16:35:22 +00:00
bp = g_alloc_bio();
bp->bio_cmd = BIO_DELETE;
bp->bio_done = NULL;
bp->bio_offset = offset;
bp->bio_length = length;
bp->bio_data = NULL;
g_io_request(bp, cp);
error = biowait(bp, "gdelete");
g_destroy_bio(bp);
return (error);
}
Added g_print_bio() function to print informations about given bio. Approved by: phk, scottl (mentor)
2004-02-11 18:21:32 +00:00
void
g_print_bio(struct bio *bp)
{
const char *pname, *cmd = NULL;
if (bp->bio_to != NULL)
pname = bp->bio_to->name;
else
pname = "[unknown]";
switch (bp->bio_cmd) {
case BIO_GETATTR:
cmd = "GETATTR";
printf("%s[%s(attr=%s)]", pname, cmd, bp->bio_attribute);
return;
Add a new I/O request - BIO_FLUSH, which basically tells providers below to flush their caches. For now will mostly be used by disks to flush their write cache. Sponsored by: home.pl
2006-10-31 21:11:21 +00:00
case BIO_FLUSH:
cmd = "FLUSH";
printf("%s[%s]", pname, cmd);
return;
Added g_print_bio() function to print informations about given bio. Approved by: phk, scottl (mentor)
2004-02-11 18:21:32 +00:00
case BIO_READ:
cmd = "READ";
Untangle g_print_bio(), silencing Coverity. Found with: Coverity Prevent CID: 3566, 3567
2010-06-10 17:49:36 +00:00
break;
Added g_print_bio() function to print informations about given bio. Approved by: phk, scottl (mentor)
2004-02-11 18:21:32 +00:00
case BIO_WRITE:
Untangle g_print_bio(), silencing Coverity. Found with: Coverity Prevent CID: 3566, 3567
2010-06-10 17:49:36 +00:00
cmd = "WRITE";
break;
Added g_print_bio() function to print informations about given bio. Approved by: phk, scottl (mentor)
2004-02-11 18:21:32 +00:00
case BIO_DELETE:
Untangle g_print_bio(), silencing Coverity. Found with: Coverity Prevent CID: 3566, 3567
2010-06-10 17:49:36 +00:00
cmd = "DELETE";
break;
Added g_print_bio() function to print informations about given bio. Approved by: phk, scottl (mentor)
2004-02-11 18:21:32 +00:00
default:
cmd = "UNKNOWN";
printf("%s[%s()]", pname, cmd);
return;
}
Untangle g_print_bio(), silencing Coverity. Found with: Coverity Prevent CID: 3566, 3567
2010-06-10 17:49:36 +00:00
printf("%s[%s(offset=%jd, length=%jd)]", pname, cmd,
(intmax_t)bp->bio_offset, (intmax_t)bp->bio_length);
Added g_print_bio() function to print informations about given bio. Approved by: phk, scottl (mentor)
2004-02-11 18:21:32 +00:00
}
Reference in New Issue Copy Permalink