2002-03-11 21:42:35 +00:00
|
|
|
/*-
|
|
|
|
|
* Copyright (c) 2002 Poul-Henning Kamp
|
|
|
|
|
* Copyright (c) 2002 Networks Associates Technology, Inc.
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
* Copyright (c) 2013 The FreeBSD Foundation
|
2002-03-11 21:42:35 +00:00
|
|
|
* All rights reserved.
|
|
|
|
|
*
|
|
|
|
|
* This software was developed for the FreeBSD Project by Poul-Henning Kamp
|
|
|
|
|
* and NAI Labs, the Security Research Division of Network Associates, Inc.
|
|
|
|
|
* under DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the
|
|
|
|
|
* DARPA CHATS research program.
|
|
|
|
|
*
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
* Portions of this software were developed by Konstantin Belousov
|
|
|
|
|
* under sponsorship from the FreeBSD Foundation.
|
|
|
|
|
*
|
2002-03-11 21:42:35 +00:00
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
|
* 3. The names of the authors may not be used to endorse or promote
|
|
|
|
|
* products derived from this software without specific prior written
|
|
|
|
|
* permission.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
|
|
|
|
|
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
|
|
|
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
|
|
|
|
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
|
|
|
|
|
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
|
|
|
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
|
|
|
|
|
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
|
|
|
|
|
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
|
|
|
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
|
|
|
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
|
|
|
|
|
* SUCH DAMAGE.
|
|
|
|
|
*/
|
|
|
|
|
|
2003-06-11 06:49:16 +00:00
|
|
|
#include <sys/cdefs.h>
|
|
|
|
|
__FBSDID("$FreeBSD$");
|
2002-03-11 21:42:35 +00:00
|
|
|
|
|
|
|
|
#include <sys/param.h>
|
|
|
|
|
#include <sys/systm.h>
|
|
|
|
|
#include <sys/kernel.h>
|
|
|
|
|
#include <sys/malloc.h>
|
|
|
|
|
#include <sys/bio.h>
|
2004-10-21 18:35:24 +00:00
|
|
|
#include <sys/ktr.h>
|
2005-09-15 19:05:37 +00:00
|
|
|
#include <sys/proc.h>
|
2005-08-29 11:39:24 +00:00
|
|
|
#include <sys/stack.h>
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
#include <sys/sysctl.h>
|
2002-03-11 21:42:35 +00:00
|
|
|
|
|
|
|
|
#include <sys/errno.h>
|
|
|
|
|
#include <geom/geom.h>
|
2002-03-26 22:07:38 +00:00
|
|
|
#include <geom/geom_int.h>
|
2003-03-18 09:42:33 +00:00
|
|
|
#include <sys/devicestat.h>
|
2002-03-11 21:42:35 +00:00
|
|
|
|
2003-05-02 12:36:12 +00:00
|
|
|
#include <vm/uma.h>
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
#include <vm/vm.h>
|
|
|
|
|
#include <vm/vm_param.h>
|
|
|
|
|
#include <vm/vm_kern.h>
|
|
|
|
|
#include <vm/vm_page.h>
|
|
|
|
|
#include <vm/vm_object.h>
|
|
|
|
|
#include <vm/vm_extern.h>
|
|
|
|
|
#include <vm/vm_map.h>
|
2003-05-02 12:36:12 +00:00
|
|
|
|
2002-03-11 21:42:35 +00:00
|
|
|
static struct g_bioq g_bio_run_down;
|
|
|
|
|
static struct g_bioq g_bio_run_up;
|
2004-01-28 08:39:18 +00:00
|
|
|
static struct g_bioq g_bio_run_task;
|
2002-03-11 21:42:35 +00:00
|
|
|
|
2002-11-02 11:08:07 +00:00
|
|
|
static u_int pace;
|
2003-05-02 12:36:12 +00:00
|
|
|
static uma_zone_t biozone;
|
2002-11-02 11:08:07 +00:00
|
|
|
|
2009-06-11 09:55:26 +00:00
|
|
|
/*
|
|
|
|
|
* The head of the list of classifiers used in g_io_request.
|
|
|
|
|
* Use g_register_classifier() and g_unregister_classifier()
|
|
|
|
|
* to add/remove entries to the list.
|
|
|
|
|
* Classifiers are invoked in registration order.
|
|
|
|
|
*/
|
|
|
|
|
static TAILQ_HEAD(g_classifier_tailq, g_classifier_hook)
|
|
|
|
|
g_classifier_tailq = TAILQ_HEAD_INITIALIZER(g_classifier_tailq);
|
|
|
|
|
|
2002-03-11 21:42:35 +00:00
|
|
|
#include <machine/atomic.h>
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
g_bioq_lock(struct g_bioq *bq)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
mtx_lock(&bq->bio_queue_lock);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
g_bioq_unlock(struct g_bioq *bq)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
mtx_unlock(&bq->bio_queue_lock);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#if 0
|
|
|
|
|
static void
|
|
|
|
|
g_bioq_destroy(struct g_bioq *bq)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
mtx_destroy(&bq->bio_queue_lock);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
g_bioq_init(struct g_bioq *bq)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
TAILQ_INIT(&bq->bio_queue);
|
2002-04-04 21:03:38 +00:00
|
|
|
mtx_init(&bq->bio_queue_lock, "bio queue", NULL, MTX_DEF);
|
2002-03-11 21:42:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static struct bio *
|
|
|
|
|
g_bioq_first(struct g_bioq *bq)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp;
|
|
|
|
|
|
|
|
|
|
bp = TAILQ_FIRST(&bq->bio_queue);
|
|
|
|
|
if (bp != NULL) {
|
2004-08-30 09:33:06 +00:00
|
|
|
KASSERT((bp->bio_flags & BIO_ONQUEUE),
|
|
|
|
|
("Bio not on queue bp=%p target %p", bp, bq));
|
|
|
|
|
bp->bio_flags &= ~BIO_ONQUEUE;
|
2002-03-11 21:42:35 +00:00
|
|
|
TAILQ_REMOVE(&bq->bio_queue, bp, bio_queue);
|
|
|
|
|
bq->bio_queue_length--;
|
|
|
|
|
}
|
|
|
|
|
return (bp);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
struct bio *
|
|
|
|
|
g_new_bio(void)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp;
|
|
|
|
|
|
2003-05-02 12:36:12 +00:00
|
|
|
bp = uma_zalloc(biozone, M_NOWAIT | M_ZERO);
|
2005-08-29 11:39:24 +00:00
|
|
|
#ifdef KTR
|
2007-10-26 06:55:00 +00:00
|
|
|
if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) {
|
2005-08-29 11:39:24 +00:00
|
|
|
struct stack st;
|
|
|
|
|
|
|
|
|
|
CTR1(KTR_GEOM, "g_new_bio(): %p", bp);
|
|
|
|
|
stack_save(&st);
|
|
|
|
|
CTRSTACK(KTR_GEOM, &st, 3, 0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
2002-03-11 21:42:35 +00:00
|
|
|
return (bp);
|
|
|
|
|
}
|
|
|
|
|
|
2004-08-27 14:43:11 +00:00
|
|
|
struct bio *
|
|
|
|
|
g_alloc_bio(void)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp;
|
|
|
|
|
|
|
|
|
|
bp = uma_zalloc(biozone, M_WAITOK | M_ZERO);
|
2005-08-29 11:39:24 +00:00
|
|
|
#ifdef KTR
|
2007-10-26 06:55:00 +00:00
|
|
|
if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) {
|
2005-08-29 11:39:24 +00:00
|
|
|
struct stack st;
|
|
|
|
|
|
|
|
|
|
CTR1(KTR_GEOM, "g_alloc_bio(): %p", bp);
|
|
|
|
|
stack_save(&st);
|
|
|
|
|
CTRSTACK(KTR_GEOM, &st, 3, 0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
2004-08-27 14:43:11 +00:00
|
|
|
return (bp);
|
|
|
|
|
}
|
|
|
|
|
|
2002-03-11 21:42:35 +00:00
|
|
|
void
|
|
|
|
|
g_destroy_bio(struct bio *bp)
|
|
|
|
|
{
|
2005-08-29 11:39:24 +00:00
|
|
|
#ifdef KTR
|
2007-10-26 06:55:00 +00:00
|
|
|
if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) {
|
2005-08-29 11:39:24 +00:00
|
|
|
struct stack st;
|
2002-03-11 21:42:35 +00:00
|
|
|
|
2005-08-29 11:39:24 +00:00
|
|
|
CTR1(KTR_GEOM, "g_destroy_bio(): %p", bp);
|
|
|
|
|
stack_save(&st);
|
|
|
|
|
CTRSTACK(KTR_GEOM, &st, 3, 0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
2003-05-02 12:36:12 +00:00
|
|
|
uma_zfree(biozone, bp);
|
2002-03-11 21:42:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
struct bio *
|
|
|
|
|
g_clone_bio(struct bio *bp)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp2;
|
|
|
|
|
|
2003-05-02 12:36:12 +00:00
|
|
|
bp2 = uma_zalloc(biozone, M_NOWAIT | M_ZERO);
|
2002-09-27 20:53:47 +00:00
|
|
|
if (bp2 != NULL) {
|
2003-02-07 21:09:51 +00:00
|
|
|
bp2->bio_parent = bp;
|
2002-09-27 20:53:47 +00:00
|
|
|
bp2->bio_cmd = bp->bio_cmd;
|
2012-08-07 20:16:10 +00:00
|
|
|
/*
|
|
|
|
|
* BIO_ORDERED flag may be used by disk drivers to enforce
|
|
|
|
|
* ordering restrictions, so this flag needs to be cloned.
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
* BIO_UNMAPPED should be inherited, to properly indicate
|
|
|
|
|
* which way the buffer is passed.
|
2012-08-07 20:16:10 +00:00
|
|
|
* Other bio flags are not suitable for cloning.
|
|
|
|
|
*/
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
bp2->bio_flags = bp->bio_flags & (BIO_ORDERED | BIO_UNMAPPED);
|
2002-09-27 20:53:47 +00:00
|
|
|
bp2->bio_length = bp->bio_length;
|
|
|
|
|
bp2->bio_offset = bp->bio_offset;
|
|
|
|
|
bp2->bio_data = bp->bio_data;
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
bp2->bio_ma = bp->bio_ma;
|
|
|
|
|
bp2->bio_ma_n = bp->bio_ma_n;
|
|
|
|
|
bp2->bio_ma_offset = bp->bio_ma_offset;
|
2002-09-27 20:53:47 +00:00
|
|
|
bp2->bio_attribute = bp->bio_attribute;
|
2009-06-11 09:55:26 +00:00
|
|
|
/* Inherit classification info from the parent */
|
|
|
|
|
bp2->bio_classifier1 = bp->bio_classifier1;
|
|
|
|
|
bp2->bio_classifier2 = bp->bio_classifier2;
|
2003-02-07 23:08:24 +00:00
|
|
|
bp->bio_children++;
|
2002-09-27 20:53:47 +00:00
|
|
|
}
|
2005-08-29 11:39:24 +00:00
|
|
|
#ifdef KTR
|
2007-10-26 06:55:00 +00:00
|
|
|
if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) {
|
2005-08-29 11:39:24 +00:00
|
|
|
struct stack st;
|
|
|
|
|
|
2006-03-13 14:59:57 +00:00
|
|
|
CTR2(KTR_GEOM, "g_clone_bio(%p): %p", bp, bp2);
|
2005-08-29 11:39:24 +00:00
|
|
|
stack_save(&st);
|
|
|
|
|
CTRSTACK(KTR_GEOM, &st, 3, 0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
2002-03-11 21:42:35 +00:00
|
|
|
return(bp2);
|
|
|
|
|
}
|
|
|
|
|
|
2006-06-05 21:13:22 +00:00
|
|
|
struct bio *
|
|
|
|
|
g_duplicate_bio(struct bio *bp)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp2;
|
|
|
|
|
|
|
|
|
|
bp2 = uma_zalloc(biozone, M_WAITOK | M_ZERO);
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
bp2->bio_flags = bp->bio_flags & BIO_UNMAPPED;
|
2006-06-05 21:13:22 +00:00
|
|
|
bp2->bio_parent = bp;
|
|
|
|
|
bp2->bio_cmd = bp->bio_cmd;
|
|
|
|
|
bp2->bio_length = bp->bio_length;
|
|
|
|
|
bp2->bio_offset = bp->bio_offset;
|
|
|
|
|
bp2->bio_data = bp->bio_data;
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
bp2->bio_ma = bp->bio_ma;
|
|
|
|
|
bp2->bio_ma_n = bp->bio_ma_n;
|
|
|
|
|
bp2->bio_ma_offset = bp->bio_ma_offset;
|
2006-06-05 21:13:22 +00:00
|
|
|
bp2->bio_attribute = bp->bio_attribute;
|
|
|
|
|
bp->bio_children++;
|
|
|
|
|
#ifdef KTR
|
2007-10-26 06:55:00 +00:00
|
|
|
if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) {
|
2006-06-05 21:13:22 +00:00
|
|
|
struct stack st;
|
|
|
|
|
|
|
|
|
|
CTR2(KTR_GEOM, "g_duplicate_bio(%p): %p", bp, bp2);
|
|
|
|
|
stack_save(&st);
|
|
|
|
|
CTRSTACK(KTR_GEOM, &st, 3, 0);
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
return(bp2);
|
|
|
|
|
}
|
|
|
|
|
|
2002-03-11 21:42:35 +00:00
|
|
|
void
|
|
|
|
|
g_io_init()
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
g_bioq_init(&g_bio_run_down);
|
|
|
|
|
g_bioq_init(&g_bio_run_up);
|
2004-01-28 08:39:18 +00:00
|
|
|
g_bioq_init(&g_bio_run_task);
|
2003-05-02 12:36:12 +00:00
|
|
|
biozone = uma_zcreate("g_bio", sizeof (struct bio),
|
|
|
|
|
NULL, NULL,
|
|
|
|
|
NULL, NULL,
|
|
|
|
|
0, 0);
|
2002-03-11 21:42:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
int
|
2002-04-09 15:12:05 +00:00
|
|
|
g_io_getattr(const char *attr, struct g_consumer *cp, int *len, void *ptr)
|
2002-03-11 21:42:35 +00:00
|
|
|
{
|
|
|
|
|
struct bio *bp;
|
|
|
|
|
int error;
|
|
|
|
|
|
|
|
|
|
g_trace(G_T_BIO, "bio_getattr(%s)", attr);
|
2004-08-27 14:43:11 +00:00
|
|
|
bp = g_alloc_bio();
|
2002-10-08 07:03:58 +00:00
|
|
|
bp->bio_cmd = BIO_GETATTR;
|
|
|
|
|
bp->bio_done = NULL;
|
|
|
|
|
bp->bio_attribute = attr;
|
|
|
|
|
bp->bio_length = *len;
|
|
|
|
|
bp->bio_data = ptr;
|
|
|
|
|
g_io_request(bp, cp);
|
|
|
|
|
error = biowait(bp, "ggetattr");
|
|
|
|
|
*len = bp->bio_completed;
|
|
|
|
|
g_destroy_bio(bp);
|
2002-03-11 21:42:35 +00:00
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
2006-10-31 21:11:21 +00:00
|
|
|
int
|
|
|
|
|
g_io_flush(struct g_consumer *cp)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp;
|
|
|
|
|
int error;
|
|
|
|
|
|
|
|
|
|
g_trace(G_T_BIO, "bio_flush(%s)", cp->provider->name);
|
|
|
|
|
bp = g_alloc_bio();
|
|
|
|
|
bp->bio_cmd = BIO_FLUSH;
|
Correct bioq_disksort so that bioq_insert_tail() offers barrier semantic.
Add the BIO_ORDERED flag for struct bio and update bio clients to use it.
The barrier semantics of bioq_insert_tail() were broken in two ways:
o In bioq_disksort(), an added bio could be inserted at the head of
the queue, even when a barrier was present, if the sort key for
the new entry was less than that of the last queued barrier bio.
o The last_offset used to generate the sort key for newly queued bios
did not stay at the position of the barrier until either the
barrier was de-queued, or a new barrier (which updates last_offset)
was queued. When a barrier is in effect, we know that the disk
will pass through the barrier position just before the
"blocked bios" are released, so using the barrier's offset for
last_offset is the optimal choice.
sys/geom/sched/subr_disk.c:
sys/kern/subr_disk.c:
o Update last_offset in bioq_insert_tail().
o Only update last_offset in bioq_remove() if the removed bio is
at the head of the queue (typically due to a call via
bioq_takefirst()) and no barrier is active.
o In bioq_disksort(), if we have a barrier (insert_point is non-NULL),
set prev to the barrier and cur to it's next element. Now that
last_offset is kept at the barrier position, this change isn't
strictly necessary, but since we have to take a decision branch
anyway, it does avoid one, no-op, loop iteration in the while
loop that immediately follows.
o In bioq_disksort(), bypass the normal sort for bios with the
BIO_ORDERED attribute and instead insert them into the queue
with bioq_insert_tail(). bioq_insert_tail() not only gives
the desired command order during insertion, but also provides
barrier semantics so that commands disksorted in the future
cannot pass the just enqueued transaction.
sys/sys/bio.h:
Add BIO_ORDERED as bit 4 of the bio_flags field in struct bio.
sys/cam/ata/ata_da.c:
sys/cam/scsi/scsi_da.c
Use an ordered command for SCSI/ATA-NCQ commands issued in
response to bios with the BIO_ORDERED flag set.
sys/cam/scsi/scsi_da.c
Use an ordered tag when issuing a synchronize cache command.
Wrap some lines to 80 columns.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c
sys/geom/geom_io.c
Mark bios with the BIO_FLUSH command as BIO_ORDERED.
Sponsored by: Spectra Logic Corporation
MFC after: 1 month
2010-09-02 19:40:28 +00:00
|
|
|
bp->bio_flags |= BIO_ORDERED;
|
2006-10-31 21:11:21 +00:00
|
|
|
bp->bio_done = NULL;
|
|
|
|
|
bp->bio_attribute = NULL;
|
|
|
|
|
bp->bio_offset = cp->provider->mediasize;
|
|
|
|
|
bp->bio_length = 0;
|
|
|
|
|
bp->bio_data = NULL;
|
|
|
|
|
g_io_request(bp, cp);
|
|
|
|
|
error = biowait(bp, "gflush");
|
|
|
|
|
g_destroy_bio(bp);
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
2003-02-06 21:01:36 +00:00
|
|
|
static int
|
|
|
|
|
g_io_check(struct bio *bp)
|
2002-03-11 21:42:35 +00:00
|
|
|
{
|
2003-02-06 21:01:36 +00:00
|
|
|
struct g_consumer *cp;
|
|
|
|
|
struct g_provider *pp;
|
2002-03-11 21:42:35 +00:00
|
|
|
|
2003-02-06 21:01:36 +00:00
|
|
|
cp = bp->bio_from;
|
|
|
|
|
pp = bp->bio_to;
|
2002-03-11 21:42:35 +00:00
|
|
|
|
2003-02-06 21:01:36 +00:00
|
|
|
/* Fail if access counters dont allow the operation */
|
2002-04-04 09:58:20 +00:00
|
|
|
switch(bp->bio_cmd) {
|
|
|
|
|
case BIO_READ:
|
|
|
|
|
case BIO_GETATTR:
|
2003-02-06 21:01:36 +00:00
|
|
|
if (cp->acr == 0)
|
|
|
|
|
return (EPERM);
|
2002-04-04 09:58:20 +00:00
|
|
|
break;
|
|
|
|
|
case BIO_WRITE:
|
|
|
|
|
case BIO_DELETE:
|
2006-10-31 21:11:21 +00:00
|
|
|
case BIO_FLUSH:
|
2003-02-06 21:01:36 +00:00
|
|
|
if (cp->acw == 0)
|
|
|
|
|
return (EPERM);
|
2002-04-04 09:58:20 +00:00
|
|
|
break;
|
|
|
|
|
default:
|
2003-02-06 21:01:36 +00:00
|
|
|
return (EPERM);
|
2002-03-11 21:42:35 +00:00
|
|
|
}
|
2002-04-04 09:58:20 +00:00
|
|
|
/* if provider is marked for error, don't disturb. */
|
2003-02-06 21:01:36 +00:00
|
|
|
if (pp->error)
|
|
|
|
|
return (pp->error);
|
2012-07-29 11:51:48 +00:00
|
|
|
if (cp->flags & G_CF_ORPHAN)
|
|
|
|
|
return (ENXIO);
|
2003-02-06 21:01:36 +00:00
|
|
|
|
2002-04-04 09:58:20 +00:00
|
|
|
switch(bp->bio_cmd) {
|
|
|
|
|
case BIO_READ:
|
|
|
|
|
case BIO_WRITE:
|
|
|
|
|
case BIO_DELETE:
|
2010-04-15 08:39:56 +00:00
|
|
|
/* Zero sectorsize or mediasize is probably a lack of media. */
|
|
|
|
|
if (pp->sectorsize == 0 || pp->mediasize == 0)
|
2003-10-22 06:32:20 +00:00
|
|
|
return (ENXIO);
|
2002-12-18 19:53:59 +00:00
|
|
|
/* Reject I/O not on sector boundary */
|
2003-02-06 21:01:36 +00:00
|
|
|
if (bp->bio_offset % pp->sectorsize)
|
|
|
|
|
return (EINVAL);
|
2002-12-18 19:53:59 +00:00
|
|
|
/* Reject I/O not integral sector long */
|
2003-02-06 21:01:36 +00:00
|
|
|
if (bp->bio_length % pp->sectorsize)
|
|
|
|
|
return (EINVAL);
|
2003-10-19 19:06:54 +00:00
|
|
|
/* Reject requests before or past the end of media. */
|
|
|
|
|
if (bp->bio_offset < 0)
|
|
|
|
|
return (EIO);
|
2003-02-06 21:01:36 +00:00
|
|
|
if (bp->bio_offset > pp->mediasize)
|
|
|
|
|
return (EIO);
|
2002-04-04 09:58:20 +00:00
|
|
|
break;
|
|
|
|
|
default:
|
|
|
|
|
break;
|
2002-03-11 21:42:35 +00:00
|
|
|
}
|
2003-02-06 21:01:36 +00:00
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
2009-06-11 09:55:26 +00:00
|
|
|
/*
|
|
|
|
|
* bio classification support.
|
|
|
|
|
*
|
|
|
|
|
* g_register_classifier() and g_unregister_classifier()
|
|
|
|
|
* are used to add/remove a classifier from the list.
|
|
|
|
|
* The list is protected using the g_bio_run_down lock,
|
|
|
|
|
* because the classifiers are called in this path.
|
|
|
|
|
*
|
|
|
|
|
* g_io_request() passes bio's that are not already classified
|
|
|
|
|
* (i.e. those with bio_classifier1 == NULL) to g_run_classifiers().
|
|
|
|
|
* Classifiers can store their result in the two fields
|
|
|
|
|
* bio_classifier1 and bio_classifier2.
|
|
|
|
|
* A classifier that updates one of the fields should
|
|
|
|
|
* return a non-zero value.
|
|
|
|
|
* If no classifier updates the field, g_run_classifiers() sets
|
|
|
|
|
* bio_classifier1 = BIO_NOTCLASSIFIED to avoid further calls.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
int
|
|
|
|
|
g_register_classifier(struct g_classifier_hook *hook)
|
|
|
|
|
{
|
|
|
|
|
|
|
|
|
|
g_bioq_lock(&g_bio_run_down);
|
|
|
|
|
TAILQ_INSERT_TAIL(&g_classifier_tailq, hook, link);
|
|
|
|
|
g_bioq_unlock(&g_bio_run_down);
|
|
|
|
|
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
|
|
|
|
g_unregister_classifier(struct g_classifier_hook *hook)
|
|
|
|
|
{
|
|
|
|
|
struct g_classifier_hook *entry;
|
|
|
|
|
|
|
|
|
|
g_bioq_lock(&g_bio_run_down);
|
|
|
|
|
TAILQ_FOREACH(entry, &g_classifier_tailq, link) {
|
|
|
|
|
if (entry == hook) {
|
|
|
|
|
TAILQ_REMOVE(&g_classifier_tailq, hook, link);
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
g_bioq_unlock(&g_bio_run_down);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void
|
|
|
|
|
g_run_classifiers(struct bio *bp)
|
|
|
|
|
{
|
|
|
|
|
struct g_classifier_hook *hook;
|
|
|
|
|
int classified = 0;
|
|
|
|
|
|
|
|
|
|
TAILQ_FOREACH(hook, &g_classifier_tailq, link)
|
|
|
|
|
classified |= hook->func(hook->arg, bp);
|
|
|
|
|
|
|
|
|
|
if (!classified)
|
|
|
|
|
bp->bio_classifier1 = BIO_NOTCLASSIFIED;
|
|
|
|
|
}
|
|
|
|
|
|
2003-02-06 21:01:36 +00:00
|
|
|
void
|
|
|
|
|
g_io_request(struct bio *bp, struct g_consumer *cp)
|
|
|
|
|
{
|
2003-02-07 23:08:24 +00:00
|
|
|
struct g_provider *pp;
|
2009-12-30 17:23:27 +00:00
|
|
|
int first;
|
2003-02-06 21:01:36 +00:00
|
|
|
|
|
|
|
|
KASSERT(cp != NULL, ("NULL cp in g_io_request"));
|
|
|
|
|
KASSERT(bp != NULL, ("NULL bp in g_io_request"));
|
2003-09-11 00:49:02 +00:00
|
|
|
pp = cp->provider;
|
2003-02-07 23:08:24 +00:00
|
|
|
KASSERT(pp != NULL, ("consumer not attached in g_io_request"));
|
2006-03-01 19:01:58 +00:00
|
|
|
#ifdef DIAGNOSTIC
|
|
|
|
|
KASSERT(bp->bio_driver1 == NULL,
|
|
|
|
|
("bio_driver1 used by the consumer (geom %s)", cp->geom->name));
|
|
|
|
|
KASSERT(bp->bio_driver2 == NULL,
|
|
|
|
|
("bio_driver2 used by the consumer (geom %s)", cp->geom->name));
|
|
|
|
|
KASSERT(bp->bio_pflags == 0,
|
|
|
|
|
("bio_pflags used by the consumer (geom %s)", cp->geom->name));
|
|
|
|
|
/*
|
|
|
|
|
* Remember consumer's private fields, so we can detect if they were
|
|
|
|
|
* modified by the provider.
|
|
|
|
|
*/
|
|
|
|
|
bp->_bio_caller1 = bp->bio_caller1;
|
|
|
|
|
bp->_bio_caller2 = bp->bio_caller2;
|
|
|
|
|
bp->_bio_cflags = bp->bio_cflags;
|
|
|
|
|
#endif
|
2003-02-07 23:08:24 +00:00
|
|
|
|
2007-01-28 23:36:07 +00:00
|
|
|
if (bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_GETATTR)) {
|
2006-10-31 21:11:21 +00:00
|
|
|
KASSERT(bp->bio_data != NULL,
|
2007-01-28 23:36:07 +00:00
|
|
|
("NULL bp->data in g_io_request(cmd=%hhu)", bp->bio_cmd));
|
|
|
|
|
}
|
|
|
|
|
if (bp->bio_cmd & (BIO_DELETE|BIO_FLUSH)) {
|
|
|
|
|
KASSERT(bp->bio_data == NULL,
|
|
|
|
|
("non-NULL bp->data in g_io_request(cmd=%hhu)",
|
|
|
|
|
bp->bio_cmd));
|
2006-10-31 21:11:21 +00:00
|
|
|
}
|
2004-08-30 09:33:06 +00:00
|
|
|
if (bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_DELETE)) {
|
|
|
|
|
KASSERT(bp->bio_offset % cp->provider->sectorsize == 0,
|
|
|
|
|
("wrong offset %jd for sectorsize %u",
|
|
|
|
|
bp->bio_offset, cp->provider->sectorsize));
|
|
|
|
|
KASSERT(bp->bio_length % cp->provider->sectorsize == 0,
|
|
|
|
|
("wrong length %jd for sectorsize %u",
|
|
|
|
|
bp->bio_length, cp->provider->sectorsize));
|
|
|
|
|
}
|
|
|
|
|
|
2004-10-11 21:22:59 +00:00
|
|
|
g_trace(G_T_BIO, "bio_request(%p) from %p(%s) to %p(%s) cmd %d",
|
|
|
|
|
bp, cp, cp->geom->name, pp, pp->name, bp->bio_cmd);
|
|
|
|
|
|
2003-02-06 21:01:36 +00:00
|
|
|
bp->bio_from = cp;
|
2003-02-07 23:08:24 +00:00
|
|
|
bp->bio_to = pp;
|
2003-02-06 21:01:36 +00:00
|
|
|
bp->bio_error = 0;
|
|
|
|
|
bp->bio_completed = 0;
|
|
|
|
|
|
2004-09-28 11:56:37 +00:00
|
|
|
KASSERT(!(bp->bio_flags & BIO_ONQUEUE),
|
|
|
|
|
("Bio already on queue bp=%p", bp));
|
|
|
|
|
bp->bio_flags |= BIO_ONQUEUE;
|
|
|
|
|
|
2010-03-24 18:04:25 +00:00
|
|
|
if (g_collectstats)
|
|
|
|
|
binuptime(&bp->bio_t0);
|
|
|
|
|
else
|
|
|
|
|
getbinuptime(&bp->bio_t0);
|
2005-07-25 21:12:54 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* The statistics collection is lockless, as such, but we
|
|
|
|
|
* can not update one instance of the statistics from more
|
|
|
|
|
* than one thread at a time, so grab the lock first.
|
2009-06-11 09:55:26 +00:00
|
|
|
*
|
|
|
|
|
* We also use the lock to protect the list of classifiers.
|
2005-07-25 21:12:54 +00:00
|
|
|
*/
|
|
|
|
|
g_bioq_lock(&g_bio_run_down);
|
2009-06-11 09:55:26 +00:00
|
|
|
|
|
|
|
|
if (!TAILQ_EMPTY(&g_classifier_tailq) && !bp->bio_classifier1)
|
|
|
|
|
g_run_classifiers(bp);
|
|
|
|
|
|
2004-06-09 19:44:44 +00:00
|
|
|
if (g_collectstats & 1)
|
2004-09-28 11:56:37 +00:00
|
|
|
devstat_start_transaction(pp->stat, &bp->bio_t0);
|
2004-06-09 19:44:44 +00:00
|
|
|
if (g_collectstats & 2)
|
2004-09-28 11:56:37 +00:00
|
|
|
devstat_start_transaction(cp->stat, &bp->bio_t0);
|
|
|
|
|
|
|
|
|
|
pp->nstart++;
|
2004-06-09 19:44:44 +00:00
|
|
|
cp->nstart++;
|
2009-12-30 17:23:27 +00:00
|
|
|
first = TAILQ_EMPTY(&g_bio_run_down.bio_queue);
|
2004-09-28 11:56:37 +00:00
|
|
|
TAILQ_INSERT_TAIL(&g_bio_run_down.bio_queue, bp, bio_queue);
|
|
|
|
|
g_bio_run_down.bio_queue_length++;
|
|
|
|
|
g_bioq_unlock(&g_bio_run_down);
|
2003-02-06 21:01:36 +00:00
|
|
|
|
2002-04-04 09:58:20 +00:00
|
|
|
/* Pass it on down. */
|
2009-12-30 17:23:27 +00:00
|
|
|
if (first)
|
|
|
|
|
wakeup(&g_wait_down);
|
2002-03-11 21:42:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void
|
2002-09-30 08:54:46 +00:00
|
|
|
g_io_deliver(struct bio *bp, int error)
|
2002-03-11 21:42:35 +00:00
|
|
|
{
|
2003-02-07 23:08:24 +00:00
|
|
|
struct g_consumer *cp;
|
|
|
|
|
struct g_provider *pp;
|
2009-12-30 17:23:27 +00:00
|
|
|
int first;
|
2002-03-11 21:42:35 +00:00
|
|
|
|
2003-09-11 00:49:02 +00:00
|
|
|
KASSERT(bp != NULL, ("NULL bp in g_io_deliver"));
|
2003-02-07 23:08:24 +00:00
|
|
|
pp = bp->bio_to;
|
2003-10-06 09:07:35 +00:00
|
|
|
KASSERT(pp != NULL, ("NULL bio_to in g_io_deliver"));
|
|
|
|
|
cp = bp->bio_from;
|
|
|
|
|
if (cp == NULL) {
|
|
|
|
|
bp->bio_error = error;
|
|
|
|
|
bp->bio_done(bp);
|
|
|
|
|
return;
|
|
|
|
|
}
|
2003-02-07 23:08:24 +00:00
|
|
|
KASSERT(cp != NULL, ("NULL bio_from in g_io_deliver"));
|
|
|
|
|
KASSERT(cp->geom != NULL, ("NULL bio_from->geom in g_io_deliver"));
|
2009-06-30 14:34:06 +00:00
|
|
|
#ifdef DIAGNOSTIC
|
|
|
|
|
/*
|
|
|
|
|
* Some classes - GJournal in particular - can modify bio's
|
|
|
|
|
* private fields while the bio is in transit; G_GEOM_VOLATILE_BIO
|
|
|
|
|
* flag means it's an expected behaviour for that particular geom.
|
|
|
|
|
*/
|
|
|
|
|
if ((cp->geom->flags & G_GEOM_VOLATILE_BIO) == 0) {
|
|
|
|
|
KASSERT(bp->bio_caller1 == bp->_bio_caller1,
|
|
|
|
|
("bio_caller1 used by the provider %s", pp->name));
|
|
|
|
|
KASSERT(bp->bio_caller2 == bp->_bio_caller2,
|
|
|
|
|
("bio_caller2 used by the provider %s", pp->name));
|
|
|
|
|
KASSERT(bp->bio_cflags == bp->_bio_cflags,
|
|
|
|
|
("bio_cflags used by the provider %s", pp->name));
|
|
|
|
|
}
|
|
|
|
|
#endif
|
2004-04-04 20:37:28 +00:00
|
|
|
KASSERT(bp->bio_completed >= 0, ("bio_completed can't be less than 0"));
|
|
|
|
|
KASSERT(bp->bio_completed <= bp->bio_length,
|
|
|
|
|
("bio_completed can't be greater than bio_length"));
|
2002-12-26 21:02:50 +00:00
|
|
|
|
2002-03-11 21:42:35 +00:00
|
|
|
g_trace(G_T_BIO,
|
2002-12-26 21:02:50 +00:00
|
|
|
"g_io_deliver(%p) from %p(%s) to %p(%s) cmd %d error %d off %jd len %jd",
|
2003-02-07 23:08:24 +00:00
|
|
|
bp, cp, cp->geom->name, pp, pp->name, bp->bio_cmd, error,
|
2002-10-20 08:45:17 +00:00
|
|
|
(intmax_t)bp->bio_offset, (intmax_t)bp->bio_length);
|
2003-02-07 23:08:24 +00:00
|
|
|
|
2004-09-28 11:56:37 +00:00
|
|
|
KASSERT(!(bp->bio_flags & BIO_ONQUEUE),
|
|
|
|
|
("Bio already on queue bp=%p", bp));
|
|
|
|
|
|
2004-08-30 09:33:06 +00:00
|
|
|
/*
|
|
|
|
|
* XXX: next two doesn't belong here
|
|
|
|
|
*/
|
2003-03-18 09:42:33 +00:00
|
|
|
bp->bio_bcount = bp->bio_length;
|
2004-06-09 19:44:44 +00:00
|
|
|
bp->bio_resid = bp->bio_bcount - bp->bio_completed;
|
2004-09-28 11:56:37 +00:00
|
|
|
|
2005-07-25 21:12:54 +00:00
|
|
|
/*
|
|
|
|
|
* The statistics collection is lockless, as such, but we
|
|
|
|
|
* can not update one instance of the statistics from more
|
|
|
|
|
* than one thread at a time, so grab the lock first.
|
|
|
|
|
*/
|
|
|
|
|
g_bioq_lock(&g_bio_run_up);
|
2004-06-09 19:44:44 +00:00
|
|
|
if (g_collectstats & 1)
|
2003-03-18 09:42:33 +00:00
|
|
|
devstat_end_transaction_bio(pp->stat, bp);
|
2004-06-09 19:44:44 +00:00
|
|
|
if (g_collectstats & 2)
|
|
|
|
|
devstat_end_transaction_bio(cp->stat, bp);
|
2005-07-25 21:12:54 +00:00
|
|
|
|
2003-03-09 09:59:48 +00:00
|
|
|
cp->nend++;
|
|
|
|
|
pp->nend++;
|
2004-09-28 11:56:37 +00:00
|
|
|
if (error != ENOMEM) {
|
|
|
|
|
bp->bio_error = error;
|
2009-12-30 17:23:27 +00:00
|
|
|
first = TAILQ_EMPTY(&g_bio_run_up.bio_queue);
|
2004-09-28 11:56:37 +00:00
|
|
|
TAILQ_INSERT_TAIL(&g_bio_run_up.bio_queue, bp, bio_queue);
|
2004-10-06 20:59:59 +00:00
|
|
|
bp->bio_flags |= BIO_ONQUEUE;
|
2004-09-28 11:56:37 +00:00
|
|
|
g_bio_run_up.bio_queue_length++;
|
|
|
|
|
g_bioq_unlock(&g_bio_run_up);
|
2009-12-30 17:23:27 +00:00
|
|
|
if (first)
|
|
|
|
|
wakeup(&g_wait_up);
|
2002-11-02 11:08:07 +00:00
|
|
|
return;
|
|
|
|
|
}
|
2004-09-28 11:56:37 +00:00
|
|
|
g_bioq_unlock(&g_bio_run_up);
|
|
|
|
|
|
|
|
|
|
if (bootverbose)
|
|
|
|
|
printf("ENOMEM %p on %p(%s)\n", bp, pp, pp->name);
|
|
|
|
|
bp->bio_children = 0;
|
|
|
|
|
bp->bio_inbed = 0;
|
2012-12-26 20:07:47 +00:00
|
|
|
bp->bio_driver1 = NULL;
|
|
|
|
|
bp->bio_driver2 = NULL;
|
|
|
|
|
bp->bio_pflags = 0;
|
2004-09-28 11:56:37 +00:00
|
|
|
g_io_request(bp, cp);
|
|
|
|
|
pace++;
|
|
|
|
|
return;
|
2002-03-11 21:42:35 +00:00
|
|
|
}
|
|
|
|
|
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
SYSCTL_DECL(_kern_geom);
|
|
|
|
|
|
|
|
|
|
static long transient_maps;
|
|
|
|
|
SYSCTL_LONG(_kern_geom, OID_AUTO, transient_maps, CTLFLAG_RD,
|
|
|
|
|
&transient_maps, 0,
|
|
|
|
|
"Total count of the transient mapping requests");
|
|
|
|
|
u_int transient_map_retries = 10;
|
|
|
|
|
SYSCTL_UINT(_kern_geom, OID_AUTO, transient_map_retries, CTLFLAG_RW,
|
|
|
|
|
&transient_map_retries, 0,
|
|
|
|
|
"Max count of retries used before giving up on creating transient map");
|
|
|
|
|
int transient_map_hard_failures;
|
|
|
|
|
SYSCTL_INT(_kern_geom, OID_AUTO, transient_map_hard_failures, CTLFLAG_RD,
|
|
|
|
|
&transient_map_hard_failures, 0,
|
|
|
|
|
"Failures to establish the transient mapping due to retry attempts "
|
|
|
|
|
"exhausted");
|
|
|
|
|
int transient_map_soft_failures;
|
|
|
|
|
SYSCTL_INT(_kern_geom, OID_AUTO, transient_map_soft_failures, CTLFLAG_RD,
|
|
|
|
|
&transient_map_soft_failures, 0,
|
|
|
|
|
"Count of retried failures to establish the transient mapping");
|
|
|
|
|
int inflight_transient_maps;
|
|
|
|
|
SYSCTL_INT(_kern_geom, OID_AUTO, inflight_transient_maps, CTLFLAG_RD,
|
|
|
|
|
&inflight_transient_maps, 0,
|
|
|
|
|
"Current count of the active transient maps");
|
|
|
|
|
|
|
|
|
|
static int
|
|
|
|
|
g_io_transient_map_bio(struct bio *bp)
|
|
|
|
|
{
|
|
|
|
|
vm_offset_t addr;
|
|
|
|
|
long size;
|
|
|
|
|
u_int retried;
|
|
|
|
|
int rv;
|
|
|
|
|
|
2013-03-21 07:26:33 +00:00
|
|
|
KASSERT(unmapped_buf_allowed, ("unmapped disabled"));
|
|
|
|
|
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
size = round_page(bp->bio_ma_offset + bp->bio_length);
|
|
|
|
|
KASSERT(size / PAGE_SIZE == bp->bio_ma_n, ("Bio too short %p", bp));
|
|
|
|
|
addr = 0;
|
|
|
|
|
retried = 0;
|
|
|
|
|
atomic_add_long(&transient_maps, 1);
|
|
|
|
|
retry:
|
|
|
|
|
vm_map_lock(bio_transient_map);
|
|
|
|
|
if (vm_map_findspace(bio_transient_map, vm_map_min(bio_transient_map),
|
|
|
|
|
size, &addr)) {
|
|
|
|
|
vm_map_unlock(bio_transient_map);
|
|
|
|
|
if (transient_map_retries != 0 &&
|
|
|
|
|
retried >= transient_map_retries) {
|
|
|
|
|
g_io_deliver(bp, EDEADLK/* XXXKIB */);
|
|
|
|
|
CTR2(KTR_GEOM, "g_down cannot map bp %p provider %s",
|
|
|
|
|
bp, bp->bio_to->name);
|
|
|
|
|
atomic_add_int(&transient_map_hard_failures, 1);
|
|
|
|
|
return (1);
|
|
|
|
|
} else {
|
|
|
|
|
/*
|
|
|
|
|
* Naive attempt to quisce the I/O to get more
|
|
|
|
|
* in-flight requests completed and defragment
|
|
|
|
|
* the bio_transient_map.
|
|
|
|
|
*/
|
|
|
|
|
CTR3(KTR_GEOM, "g_down retrymap bp %p provider %s r %d",
|
|
|
|
|
bp, bp->bio_to->name, retried);
|
|
|
|
|
pause("g_d_tra", hz / 10);
|
|
|
|
|
retried++;
|
|
|
|
|
atomic_add_int(&transient_map_soft_failures, 1);
|
|
|
|
|
goto retry;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
rv = vm_map_insert(bio_transient_map, NULL, 0, addr, addr + size,
|
|
|
|
|
VM_PROT_RW, VM_PROT_RW, MAP_NOFAULT);
|
|
|
|
|
KASSERT(rv == KERN_SUCCESS,
|
|
|
|
|
("vm_map_insert(bio_transient_map) rv %d %jx %lx",
|
|
|
|
|
rv, (uintmax_t)addr, size));
|
|
|
|
|
vm_map_unlock(bio_transient_map);
|
|
|
|
|
atomic_add_int(&inflight_transient_maps, 1);
|
|
|
|
|
pmap_qenter((vm_offset_t)addr, bp->bio_ma, OFF_TO_IDX(size));
|
|
|
|
|
bp->bio_data = (caddr_t)addr + bp->bio_ma_offset;
|
|
|
|
|
bp->bio_flags |= BIO_TRANSIENT_MAPPING;
|
|
|
|
|
bp->bio_flags &= ~BIO_UNMAPPED;
|
|
|
|
|
return (0);
|
|
|
|
|
}
|
|
|
|
|
|
2002-03-11 21:42:35 +00:00
|
|
|
void
|
|
|
|
|
g_io_schedule_down(struct thread *tp __unused)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp;
|
2003-02-06 21:01:36 +00:00
|
|
|
off_t excess;
|
|
|
|
|
int error;
|
2002-03-11 21:42:35 +00:00
|
|
|
|
|
|
|
|
for(;;) {
|
2003-02-11 22:30:26 +00:00
|
|
|
g_bioq_lock(&g_bio_run_down);
|
2002-03-11 21:42:35 +00:00
|
|
|
bp = g_bioq_first(&g_bio_run_down);
|
2003-02-11 22:30:26 +00:00
|
|
|
if (bp == NULL) {
|
2004-10-21 18:35:24 +00:00
|
|
|
CTR0(KTR_GEOM, "g_down going to sleep");
|
2003-02-11 22:30:26 +00:00
|
|
|
msleep(&g_wait_down, &g_bio_run_down.bio_queue_lock,
|
2009-09-06 19:33:13 +00:00
|
|
|
PRIBIO | PDROP, "-", 0);
|
2003-02-11 22:30:26 +00:00
|
|
|
continue;
|
|
|
|
|
}
|
2004-10-21 18:35:24 +00:00
|
|
|
CTR0(KTR_GEOM, "g_down has work to do");
|
2003-02-11 22:30:26 +00:00
|
|
|
g_bioq_unlock(&g_bio_run_down);
|
2003-03-29 22:34:37 +00:00
|
|
|
if (pace > 0) {
|
2004-10-21 18:35:24 +00:00
|
|
|
CTR1(KTR_GEOM, "g_down pacing self (pace %d)", pace);
|
2007-02-27 17:23:29 +00:00
|
|
|
pause("g_down", hz/10);
|
2003-03-29 22:34:37 +00:00
|
|
|
pace--;
|
|
|
|
|
}
|
2003-02-06 21:01:36 +00:00
|
|
|
error = g_io_check(bp);
|
|
|
|
|
if (error) {
|
2004-10-21 18:35:24 +00:00
|
|
|
CTR3(KTR_GEOM, "g_down g_io_check on bp %p provider "
|
|
|
|
|
"%s returned %d", bp, bp->bio_to->name, error);
|
2003-02-06 21:01:36 +00:00
|
|
|
g_io_deliver(bp, error);
|
|
|
|
|
continue;
|
|
|
|
|
}
|
2004-10-21 18:35:24 +00:00
|
|
|
CTR2(KTR_GEOM, "g_down processing bp %p provider %s", bp,
|
|
|
|
|
bp->bio_to->name);
|
2003-02-11 13:13:10 +00:00
|
|
|
switch (bp->bio_cmd) {
|
|
|
|
|
case BIO_READ:
|
|
|
|
|
case BIO_WRITE:
|
|
|
|
|
case BIO_DELETE:
|
|
|
|
|
/* Truncate requests to the end of providers media. */
|
2004-10-21 18:35:24 +00:00
|
|
|
/*
|
|
|
|
|
* XXX: What if we truncate because of offset being
|
|
|
|
|
* bad, not length?
|
|
|
|
|
*/
|
2003-02-11 13:13:10 +00:00
|
|
|
excess = bp->bio_offset + bp->bio_length;
|
|
|
|
|
if (excess > bp->bio_to->mediasize) {
|
2013-03-21 22:36:43 +00:00
|
|
|
KASSERT((bp->bio_flags & BIO_UNMAPPED) == 0 ||
|
|
|
|
|
round_page(bp->bio_ma_offset +
|
|
|
|
|
bp->bio_length) / PAGE_SIZE == bp->bio_ma_n,
|
|
|
|
|
("excess bio %p too short", bp));
|
2003-02-11 13:13:10 +00:00
|
|
|
excess -= bp->bio_to->mediasize;
|
|
|
|
|
bp->bio_length -= excess;
|
2013-03-21 22:36:43 +00:00
|
|
|
if ((bp->bio_flags & BIO_UNMAPPED) != 0) {
|
|
|
|
|
bp->bio_ma_n = round_page(
|
|
|
|
|
bp->bio_ma_offset +
|
|
|
|
|
bp->bio_length) / PAGE_SIZE;
|
|
|
|
|
}
|
2004-10-21 18:35:24 +00:00
|
|
|
if (excess > 0)
|
|
|
|
|
CTR3(KTR_GEOM, "g_down truncated bio "
|
|
|
|
|
"%p provider %s by %d", bp,
|
|
|
|
|
bp->bio_to->name, excess);
|
2003-02-11 13:13:10 +00:00
|
|
|
}
|
|
|
|
|
/* Deliver zero length transfers right here. */
|
|
|
|
|
if (bp->bio_length == 0) {
|
|
|
|
|
g_io_deliver(bp, 0);
|
2004-10-21 18:35:24 +00:00
|
|
|
CTR2(KTR_GEOM, "g_down terminated 0-length "
|
|
|
|
|
"bp %p provider %s", bp, bp->bio_to->name);
|
2003-02-11 13:13:10 +00:00
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
break;
|
|
|
|
|
default:
|
|
|
|
|
break;
|
2003-02-06 21:01:36 +00:00
|
|
|
}
|
Implement the concept of the unmapped VMIO buffers, i.e. buffers which
do not map the b_pages pages into buffer_map KVA. The use of the
unmapped buffers eliminate the need to perform TLB shootdown for
mapping on the buffer creation and reuse, greatly reducing the amount
of IPIs for shootdown on big-SMP machines and eliminating up to 25-30%
of the system time on i/o intensive workloads.
The unmapped buffer should be explicitely requested by the GB_UNMAPPED
flag by the consumer. For unmapped buffer, no KVA reservation is
performed at all. The consumer might request unmapped buffer which
does have a KVA reserve, to manually map it without recursing into
buffer cache and blocking, with the GB_KVAALLOC flag.
When the mapped buffer is requested and unmapped buffer already
exists, the cache performs an upgrade, possibly reusing the KVA
reservation.
Unmapped buffer is translated into unmapped bio in g_vfs_strategy().
Unmapped bio carry a pointer to the vm_page_t array, offset and length
instead of the data pointer. The provider which processes the bio
should explicitely specify a readiness to accept unmapped bio,
otherwise g_down geom thread performs the transient upgrade of the bio
request by mapping the pages into the new bio_transient_map KVA
submap.
The bio_transient_map submap claims up to 10% of the buffer map, and
the total buffer_map + bio_transient_map KVA usage stays the
same. Still, it could be manually tuned by kern.bio_transient_maxcnt
tunable, in the units of the transient mappings. Eventually, the
bio_transient_map could be removed after all geom classes and drivers
can accept unmapped i/o requests.
Unmapped support can be turned off by the vfs.unmapped_buf_allowed
tunable, disabling which makes the buffer (or cluster) creation
requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped
buffers are only enabled by default on the architectures where
pmap_copy_page() was implemented and tested.
In the rework, filesystem metadata is not the subject to maxbufspace
limit anymore. Since the metadata buffers are always mapped, the
buffers still have to fit into the buffer map, which provides a
reasonable (but practically unreachable) upper bound on it. The
non-metadata buffer allocations, both mapped and unmapped, is
accounted against maxbufspace, as before. Effectively, this means that
the maxbufspace is forced on mapped and unmapped buffers separately.
The pre-patch bufspace limiting code did not worked, because
buffer_map fragmentation does not allow the limit to be reached.
By Jeff Roberson request, the getnewbuf() function was split into
smaller single-purpose functions.
Sponsored by: The FreeBSD Foundation
Discussed with: jeff (previous version)
Tested by: pho, scottl (previous version), jhb, bf
MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
|
|
|
if ((bp->bio_flags & BIO_UNMAPPED) != 0 &&
|
|
|
|
|
(bp->bio_to->flags & G_PF_ACCEPT_UNMAPPED) == 0 &&
|
|
|
|
|
(bp->bio_cmd == BIO_READ || bp->bio_cmd == BIO_WRITE)) {
|
|
|
|
|
if (g_io_transient_map_bio(bp))
|
|
|
|
|
continue;
|
|
|
|
|
}
|
2005-09-15 19:05:37 +00:00
|
|
|
THREAD_NO_SLEEPING();
|
2004-10-21 18:35:24 +00:00
|
|
|
CTR4(KTR_GEOM, "g_down starting bp %p provider %s off %ld "
|
|
|
|
|
"len %ld", bp, bp->bio_to->name, bp->bio_offset,
|
|
|
|
|
bp->bio_length);
|
2002-03-11 21:42:35 +00:00
|
|
|
bp->bio_to->geom->start(bp);
|
2005-09-15 19:05:37 +00:00
|
|
|
THREAD_SLEEPING_OK();
|
2002-03-11 21:42:35 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2004-01-28 08:39:18 +00:00
|
|
|
void
|
|
|
|
|
bio_taskqueue(struct bio *bp, bio_task_t *func, void *arg)
|
|
|
|
|
{
|
|
|
|
|
bp->bio_task = func;
|
|
|
|
|
bp->bio_task_arg = arg;
|
|
|
|
|
/*
|
|
|
|
|
* The taskqueue is actually just a second queue off the "up"
|
|
|
|
|
* queue, so we use the same lock.
|
|
|
|
|
*/
|
|
|
|
|
g_bioq_lock(&g_bio_run_up);
|
2004-08-30 09:33:06 +00:00
|
|
|
KASSERT(!(bp->bio_flags & BIO_ONQUEUE),
|
|
|
|
|
("Bio already on queue bp=%p target taskq", bp));
|
|
|
|
|
bp->bio_flags |= BIO_ONQUEUE;
|
2004-01-28 08:39:18 +00:00
|
|
|
TAILQ_INSERT_TAIL(&g_bio_run_task.bio_queue, bp, bio_queue);
|
|
|
|
|
g_bio_run_task.bio_queue_length++;
|
|
|
|
|
wakeup(&g_wait_up);
|
|
|
|
|
g_bioq_unlock(&g_bio_run_up);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
2002-03-11 21:42:35 +00:00
|
|
|
void
|
|
|
|
|
g_io_schedule_up(struct thread *tp __unused)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp;
|
|
|
|
|
for(;;) {
|
2003-02-11 22:30:26 +00:00
|
|
|
g_bioq_lock(&g_bio_run_up);
|
2004-01-28 08:39:18 +00:00
|
|
|
bp = g_bioq_first(&g_bio_run_task);
|
|
|
|
|
if (bp != NULL) {
|
|
|
|
|
g_bioq_unlock(&g_bio_run_up);
|
2005-09-15 19:05:37 +00:00
|
|
|
THREAD_NO_SLEEPING();
|
2004-10-21 18:35:24 +00:00
|
|
|
CTR1(KTR_GEOM, "g_up processing task bp %p", bp);
|
2004-01-28 08:39:18 +00:00
|
|
|
bp->bio_task(bp->bio_task_arg);
|
2005-09-15 19:05:37 +00:00
|
|
|
THREAD_SLEEPING_OK();
|
2004-01-28 08:39:18 +00:00
|
|
|
continue;
|
|
|
|
|
}
|
2002-03-11 21:42:35 +00:00
|
|
|
bp = g_bioq_first(&g_bio_run_up);
|
2003-02-11 22:30:26 +00:00
|
|
|
if (bp != NULL) {
|
|
|
|
|
g_bioq_unlock(&g_bio_run_up);
|
2005-09-15 19:05:37 +00:00
|
|
|
THREAD_NO_SLEEPING();
|
2004-10-21 18:35:24 +00:00
|
|
|
CTR4(KTR_GEOM, "g_up biodone bp %p provider %s off "
|
2008-09-18 15:02:19 +00:00
|
|
|
"%jd len %ld", bp, bp->bio_to->name,
|
2004-10-21 18:35:24 +00:00
|
|
|
bp->bio_offset, bp->bio_length);
|
2003-02-11 22:30:26 +00:00
|
|
|
biodone(bp);
|
2005-09-15 19:05:37 +00:00
|
|
|
THREAD_SLEEPING_OK();
|
2003-02-11 22:30:26 +00:00
|
|
|
continue;
|
|
|
|
|
}
|
2004-10-21 18:35:24 +00:00
|
|
|
CTR0(KTR_GEOM, "g_up going to sleep");
|
2003-02-11 22:30:26 +00:00
|
|
|
msleep(&g_wait_up, &g_bio_run_up.bio_queue_lock,
|
2009-09-06 19:33:13 +00:00
|
|
|
PRIBIO | PDROP, "-", 0);
|
2002-03-11 21:42:35 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void *
|
|
|
|
|
g_read_data(struct g_consumer *cp, off_t offset, off_t length, int *error)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp;
|
|
|
|
|
void *ptr;
|
|
|
|
|
int errorc;
|
|
|
|
|
|
2004-09-28 08:34:27 +00:00
|
|
|
KASSERT(length > 0 && length >= cp->provider->sectorsize &&
|
|
|
|
|
length <= MAXPHYS, ("g_read_data(): invalid length %jd",
|
|
|
|
|
(intmax_t)length));
|
2003-09-26 20:52:46 +00:00
|
|
|
|
2004-08-27 14:43:11 +00:00
|
|
|
bp = g_alloc_bio();
|
2002-10-08 07:03:58 +00:00
|
|
|
bp->bio_cmd = BIO_READ;
|
|
|
|
|
bp->bio_done = NULL;
|
|
|
|
|
bp->bio_offset = offset;
|
|
|
|
|
bp->bio_length = length;
|
2003-02-19 05:47:46 +00:00
|
|
|
ptr = g_malloc(length, M_WAITOK);
|
2002-10-08 07:03:58 +00:00
|
|
|
bp->bio_data = ptr;
|
|
|
|
|
g_io_request(bp, cp);
|
|
|
|
|
errorc = biowait(bp, "gread");
|
|
|
|
|
if (error != NULL)
|
|
|
|
|
*error = errorc;
|
|
|
|
|
g_destroy_bio(bp);
|
|
|
|
|
if (errorc) {
|
|
|
|
|
g_free(ptr);
|
|
|
|
|
ptr = NULL;
|
|
|
|
|
}
|
2002-03-11 21:42:35 +00:00
|
|
|
return (ptr);
|
|
|
|
|
}
|
2002-09-30 08:50:47 +00:00
|
|
|
|
|
|
|
|
int
|
|
|
|
|
g_write_data(struct g_consumer *cp, off_t offset, void *ptr, off_t length)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp;
|
|
|
|
|
int error;
|
|
|
|
|
|
2004-09-28 08:34:27 +00:00
|
|
|
KASSERT(length > 0 && length >= cp->provider->sectorsize &&
|
|
|
|
|
length <= MAXPHYS, ("g_write_data(): invalid length %jd",
|
|
|
|
|
(intmax_t)length));
|
2003-09-26 20:52:46 +00:00
|
|
|
|
2004-08-27 14:43:11 +00:00
|
|
|
bp = g_alloc_bio();
|
2002-09-30 08:50:47 +00:00
|
|
|
bp->bio_cmd = BIO_WRITE;
|
|
|
|
|
bp->bio_done = NULL;
|
|
|
|
|
bp->bio_offset = offset;
|
|
|
|
|
bp->bio_length = length;
|
|
|
|
|
bp->bio_data = ptr;
|
|
|
|
|
g_io_request(bp, cp);
|
|
|
|
|
error = biowait(bp, "gwrite");
|
|
|
|
|
g_destroy_bio(bp);
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
2004-02-11 18:21:32 +00:00
|
|
|
|
2007-05-05 16:35:22 +00:00
|
|
|
int
|
|
|
|
|
g_delete_data(struct g_consumer *cp, off_t offset, off_t length)
|
|
|
|
|
{
|
|
|
|
|
struct bio *bp;
|
|
|
|
|
int error;
|
|
|
|
|
|
2007-12-16 18:03:31 +00:00
|
|
|
KASSERT(length > 0 && length >= cp->provider->sectorsize,
|
|
|
|
|
("g_delete_data(): invalid length %jd", (intmax_t)length));
|
2007-05-05 16:35:22 +00:00
|
|
|
|
|
|
|
|
bp = g_alloc_bio();
|
|
|
|
|
bp->bio_cmd = BIO_DELETE;
|
|
|
|
|
bp->bio_done = NULL;
|
|
|
|
|
bp->bio_offset = offset;
|
|
|
|
|
bp->bio_length = length;
|
|
|
|
|
bp->bio_data = NULL;
|
|
|
|
|
g_io_request(bp, cp);
|
|
|
|
|
error = biowait(bp, "gdelete");
|
|
|
|
|
g_destroy_bio(bp);
|
|
|
|
|
return (error);
|
|
|
|
|
}
|
|
|
|
|
|
2004-02-11 18:21:32 +00:00
|
|
|
void
|
|
|
|
|
g_print_bio(struct bio *bp)
|
|
|
|
|
{
|
|
|
|
|
const char *pname, *cmd = NULL;
|
|
|
|
|
|
|
|
|
|
if (bp->bio_to != NULL)
|
|
|
|
|
pname = bp->bio_to->name;
|
|
|
|
|
else
|
|
|
|
|
pname = "[unknown]";
|
|
|
|
|
|
|
|
|
|
switch (bp->bio_cmd) {
|
|
|
|
|
case BIO_GETATTR:
|
|
|
|
|
cmd = "GETATTR";
|
|
|
|
|
printf("%s[%s(attr=%s)]", pname, cmd, bp->bio_attribute);
|
|
|
|
|
return;
|
2006-10-31 21:11:21 +00:00
|
|
|
case BIO_FLUSH:
|
|
|
|
|
cmd = "FLUSH";
|
|
|
|
|
printf("%s[%s]", pname, cmd);
|
|
|
|
|
return;
|
2004-02-11 18:21:32 +00:00
|
|
|
case BIO_READ:
|
|
|
|
|
cmd = "READ";
|
2010-06-10 17:49:36 +00:00
|
|
|
break;
|
2004-02-11 18:21:32 +00:00
|
|
|
case BIO_WRITE:
|
2010-06-10 17:49:36 +00:00
|
|
|
cmd = "WRITE";
|
|
|
|
|
break;
|
2004-02-11 18:21:32 +00:00
|
|
|
case BIO_DELETE:
|
2010-06-10 17:49:36 +00:00
|
|
|
cmd = "DELETE";
|
|
|
|
|
break;
|
2004-02-11 18:21:32 +00:00
|
|
|
default:
|
|
|
|
|
cmd = "UNKNOWN";
|
|
|
|
|
printf("%s[%s()]", pname, cmd);
|
|
|
|
|
return;
|
|
|
|
|
}
|
2010-06-10 17:49:36 +00:00
|
|
|
printf("%s[%s(offset=%jd, length=%jd)]", pname, cmd,
|
|
|
|
|
(intmax_t)bp->bio_offset, (intmax_t)bp->bio_length);
|
2004-02-11 18:21:32 +00:00
|
|
|
}
|
