d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2ea1fcce34
freebsd-skq/sys/geom/geom_vfs.c

285 lines
6.6 KiB
C
Raw Normal View History

Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
/*-
* Copyright (c) 2004 Poul-Henning Kamp
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/bio.h>
#include <sys/kernel.h>
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
#include <sys/lock.h>
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
#include <sys/malloc.h>
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
#include <sys/mutex.h>
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
#include <sys/vnode.h>
Remove the support for using non-mpsafe filesystem modules. In particular, do not lock Giant conditionally when calling into the filesystem module, remove the VFS_LOCK_GIANT() and related macros. Stop handling buffers belonging to non-mpsafe filesystems. The VFS_VERSION is bumped to indicate the interface change which does not result in the interface signatures changes. Conducted and reviewed by: attilio Tested by: pho
2012-10-22 17:50:54 +00:00
#include <sys/mount.h>
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
#include <geom/geom.h>
#include <geom/geom_vfs.h>
/*
* subroutines for use by filesystems.
*
* XXX: should maybe live somewhere else ?
*/
#include <sys/buf.h>
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
struct g_vfs_softc {
struct mtx sc_mtx;
struct bufobj *sc_bo;
int sc_active;
int sc_orphaned;
};
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
static struct buf_ops __g_vfs_bufops = {
.bop_name = "GEOM_VFS",
.bop_write = bufwrite,
.bop_strategy = g_vfs_strategy,
Add BO_SYNC() and add a default which uses the secret vnode pointer and VOP_FSYNC() for now.
2005-01-11 10:43:08 +00:00
.bop_sync = bufsync,
Cylinder group bitmaps and blocks containing inode for a snapshot file are after snaplock, while other ffs device buffers are before snaplock in global lock order. By itself, this could cause deadlock when bdwrite() tries to flush dirty buffers on snapshotted ffs. If, during the flush, COW activity for snapshot needs to allocate block and ffs_alloccg() selects the cylinder group that is being written by bdwrite(), then kernel would panic due to recursive buffer lock acquision. Avoid dealing with buffers in bdwrite() that are from other side of snaplock divisor in the lock order then the buffer being written. Add new BOP, bop_bdwrite(), to do dirty buffer flushing for same vnode in the bdwrite(). Default implementation, bufbdflush(), refactors the code from bdwrite(). For ffs device buffers, specialized implementation is used. Reviewed by: tegge, jeff, Russell Cattelan (cattelan xfs org, xfs changes) Tested by: Peter Holm X-MFC after: 3 weeks (if ever: it changes ABI)
2007-01-23 10:01:19 +00:00
.bop_bdflush = bufbdflush
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
};
struct buf_ops *g_vfs_bufops = &__g_vfs_bufops;
Make various random things static
2005-02-10 12:10:35 +00:00
static g_orphan_t g_vfs_orphan;
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
static struct g_class g_vfs_class = {
.name = "VFS",
.version = G_VERSION,
.orphan = g_vfs_orphan,
};
DECLARE_GEOM_CLASS(g_vfs_class, g_vfs);
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
static void
g_vfs_destroy(void *arg, int flags __unused)
{
struct g_consumer *cp;
g_topology_assert();
cp = arg;
if (cp->acr > 0 || cp->acw > 0 || cp->ace > 0)
g_access(cp, -cp->acr, -cp->acw, -cp->ace);
g_detach(cp);
if (cp->geom->softc == NULL)
g_wither_geom(cp->geom, ENXIO);
}
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
static void
g_vfs_done(struct bio *bip)
{
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
struct g_consumer *cp;
struct g_vfs_softc *sc;
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
struct buf *bp;
Remove the support for using non-mpsafe filesystem modules. In particular, do not lock Giant conditionally when calling into the filesystem module, remove the VFS_LOCK_GIANT() and related macros. Stop handling buffers belonging to non-mpsafe filesystems. The VFS_VERSION is bumped to indicate the interface change which does not result in the interface signatures changes. Conducted and reviewed by: attilio Tested by: pho
2012-10-22 17:50:54 +00:00
int destroy;
Keep track of the mount point associated with a special device to enable the collection of counts of synchronous and asynchronous reads and writes for its associated filesystem. The counts are displayed using `mount -v'. Ensure that buffers used for paging indicate the vnode from which they are operating so that counts of paging I/O operations from the filesystem are collected. This checkin only adds the setting of the mount point for the UFS/FFS filesystem, but it would be trivial to add the setting and clearing of the mount point at filesystem mount/unmount time for other filesystems too. Reviewed by: kib
2012-03-28 20:49:11 +00:00
struct mount *mp;
struct vnode *vp;
Expand locking around identification of filesystem mount point when accounting for I/O counts at completion of I/O operation. Also switch from using global devmtx to vnode mutex to reduce contention. Suggested and reviewed by: kib
2012-04-08 06:20:21 +00:00
struct cdev *cdevp;
Keep track of the mount point associated with a special device to enable the collection of counts of synchronous and asynchronous reads and writes for its associated filesystem. The counts are displayed using `mount -v'. Ensure that buffers used for paging indicate the vnode from which they are operating so that counts of paging I/O operations from the filesystem are collected. This checkin only adds the setting of the mount point for the UFS/FFS filesystem, but it would be trivial to add the setting and clearing of the mount point at filesystem mount/unmount time for other filesystems too. Reviewed by: kib
2012-03-28 20:49:11 +00:00
/*
* Collect statistics on synchronous and asynchronous read
* and write counts for disks that have associated filesystems.
*/
bp = bip->bio_caller2;
vp = bp->b_vp;
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
if (vp != NULL) {
Expand locking around identification of filesystem mount point when accounting for I/O counts at completion of I/O operation. Also switch from using global devmtx to vnode mutex to reduce contention. Suggested and reviewed by: kib
2012-04-08 06:20:21 +00:00
/*
* If not a disk vnode, use its associated mount point
* otherwise use the mountpoint associated with the disk.
*/
VI_LOCK(vp);
if (vp->v_type != VCHR ||
(cdevp = vp->v_rdev) == NULL ||
cdevp->si_devsw == NULL ||
(cdevp->si_devsw->d_flags & D_DISK) == 0)
mp = vp->v_mount;
else
mp = cdevp->si_mountpt;
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
if (mp != NULL) {
if (bp->b_iocmd == BIO_READ) {
if (LK_HOLDER(bp->b_lock.lk_lock) == LK_KERNPROC)
mp->mnt_stat.f_asyncreads++;
else
mp->mnt_stat.f_syncreads++;
} else if (bp->b_iocmd == BIO_WRITE) {
if (LK_HOLDER(bp->b_lock.lk_lock) == LK_KERNPROC)
mp->mnt_stat.f_asyncwrites++;
else
mp->mnt_stat.f_syncwrites++;
}
Keep track of the mount point associated with a special device to enable the collection of counts of synchronous and asynchronous reads and writes for its associated filesystem. The counts are displayed using `mount -v'. Ensure that buffers used for paging indicate the vnode from which they are operating so that counts of paging I/O operations from the filesystem are collected. This checkin only adds the setting of the mount point for the UFS/FFS filesystem, but it would be trivial to add the setting and clearing of the mount point at filesystem mount/unmount time for other filesystems too. Reviewed by: kib
2012-03-28 20:49:11 +00:00
}
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
VI_UNLOCK(vp);
Keep track of the mount point associated with a special device to enable the collection of counts of synchronous and asynchronous reads and writes for its associated filesystem. The counts are displayed using `mount -v'. Ensure that buffers used for paging indicate the vnode from which they are operating so that counts of paging I/O operations from the filesystem are collected. This checkin only adds the setting of the mount point for the UFS/FFS filesystem, but it would be trivial to add the setting and clearing of the mount point at filesystem mount/unmount time for other filesystems too. Reviewed by: kib
2012-03-28 20:49:11 +00:00
}
Prevent a panic that happens on SMP machines when removing a disk with many writes queued up. Reviewed by: phk, scottl Approved by: rwatson (mentor) Sponsored by: FreeBSD Foundation
2009-01-11 13:51:04 +00:00
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
cp = bip->bio_from;
sc = cp->geom->softc;
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
if (bip->bio_error) {
Finish cut&paste adjustments. Spotted by: tegge
2004-11-04 07:17:08 +00:00
printf("g_vfs_done():");
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
g_print_bio(bip);
printf("error = %d\n", bip->bio_error);
}
bp->b_error = bip->bio_error;
bp->b_ioflags = bip->bio_flags;
if (bip->bio_error)
bp->b_ioflags |= BIO_ERROR;
bp->b_resid = bp->b_bcount - bip->bio_completed;
g_destroy_bio(bip);
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
mtx_lock(&sc->sc_mtx);
destroy = ((--sc->sc_active) == 0 && sc->sc_orphaned);
mtx_unlock(&sc->sc_mtx);
if (destroy)
g_post_event(g_vfs_destroy, cp, M_WAITOK, NULL);
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
bufdone(bp);
}
void
g_vfs_strategy(struct bufobj *bo, struct buf *bp)
{
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
struct g_vfs_softc *sc;
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
struct g_consumer *cp;
struct bio *bip;
cp = bo->bo_private;
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
sc = cp->geom->softc;
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
Implement g_vfs_orphan(). Without it, the filesystem never closes the device, which means refcount on periph drivers never drops, which means cam_sim_free() never returns, which results in umass sleeping there ad infinitum. Submitted by: pjd Reviewed by: scottl, pjd Approved by: rwatson (mentor) Sponsored by: FreeBSD Foundation
2008-12-16 17:04:52 +00:00
/*
Fix typos - remove duplicate "the". PR: bin/154928 Submitted by: Eitan Adler <lists at eitanadler.com> MFC after: 3 days
2011-02-21 09:01:34 +00:00
* If the provider has orphaned us, just return EXIO.
Implement g_vfs_orphan(). Without it, the filesystem never closes the device, which means refcount on periph drivers never drops, which means cam_sim_free() never returns, which results in umass sleeping there ad infinitum. Submitted by: pjd Reviewed by: scottl, pjd Approved by: rwatson (mentor) Sponsored by: FreeBSD Foundation
2008-12-16 17:04:52 +00:00
*/
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
mtx_lock(&sc->sc_mtx);
if (sc->sc_orphaned) {
mtx_unlock(&sc->sc_mtx);
Implement g_vfs_orphan(). Without it, the filesystem never closes the device, which means refcount on periph drivers never drops, which means cam_sim_free() never returns, which results in umass sleeping there ad infinitum. Submitted by: pjd Reviewed by: scottl, pjd Approved by: rwatson (mentor) Sponsored by: FreeBSD Foundation
2008-12-16 17:04:52 +00:00
bp->b_error = ENXIO;
bp->b_ioflags |= BIO_ERROR;
bufdone(bp);
return;
}
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
sc->sc_active++;
mtx_unlock(&sc->sc_mtx);
Implement g_vfs_orphan(). Without it, the filesystem never closes the device, which means refcount on periph drivers never drops, which means cam_sim_free() never returns, which results in umass sleeping there ad infinitum. Submitted by: pjd Reviewed by: scottl, pjd Approved by: rwatson (mentor) Sponsored by: FreeBSD Foundation
2008-12-16 17:04:52 +00:00
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
bip = g_alloc_bio();
bip->bio_cmd = bp->b_iocmd;
bip->bio_offset = bp->b_iooffset;
bip->bio_length = bp->b_bcount;
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
bdata2bio(bp, bip);
if ((bp->b_flags & B_BARRIER) != 0) {
Add barrier write capability to the VFS buffer interface. A barrier write is a disk write request that tells the disk that the buffer being written must be committed to the media along with any writes that preceeded it before any future blocks may be written to the drive. Barrier writes are provided by adding the functions bbarrierwrite (bwrite with barrier) and babarrierwrite (bawrite with barrier). Following a bbarrierwrite the client knows that the requested buffer is on the media. It does not ensure that buffers written before that buffer are on the media. It only ensure that buffers written before that buffer will get to the media before any buffers written after that buffer. A flush command must be sent to the disk to ensure that all earlier written buffers are on the media. Reviewed by: kib Tested by: Peter Holm
2013-02-16 14:51:30 +00:00
bip->bio_flags |= BIO_ORDERED;
bp->b_flags &= ~B_BARRIER;
}
Implement the concept of the unmapped VMIO buffers, i.e. buffers which do not map the b_pages pages into buffer_map KVA. The use of the unmapped buffers eliminate the need to perform TLB shootdown for mapping on the buffer creation and reuse, greatly reducing the amount of IPIs for shootdown on big-SMP machines and eliminating up to 25-30% of the system time on i/o intensive workloads. The unmapped buffer should be explicitely requested by the GB_UNMAPPED flag by the consumer. For unmapped buffer, no KVA reservation is performed at all. The consumer might request unmapped buffer which does have a KVA reserve, to manually map it without recursing into buffer cache and blocking, with the GB_KVAALLOC flag. When the mapped buffer is requested and unmapped buffer already exists, the cache performs an upgrade, possibly reusing the KVA reservation. Unmapped buffer is translated into unmapped bio in g_vfs_strategy(). Unmapped bio carry a pointer to the vm_page_t array, offset and length instead of the data pointer. The provider which processes the bio should explicitely specify a readiness to accept unmapped bio, otherwise g_down geom thread performs the transient upgrade of the bio request by mapping the pages into the new bio_transient_map KVA submap. The bio_transient_map submap claims up to 10% of the buffer map, and the total buffer_map + bio_transient_map KVA usage stays the same. Still, it could be manually tuned by kern.bio_transient_maxcnt tunable, in the units of the transient mappings. Eventually, the bio_transient_map could be removed after all geom classes and drivers can accept unmapped i/o requests. Unmapped support can be turned off by the vfs.unmapped_buf_allowed tunable, disabling which makes the buffer (or cluster) creation requests to ignore GB_UNMAPPED and GB_KVAALLOC flags. Unmapped buffers are only enabled by default on the architectures where pmap_copy_page() was implemented and tested. In the rework, filesystem metadata is not the subject to maxbufspace limit anymore. Since the metadata buffers are always mapped, the buffers still have to fit into the buffer map, which provides a reasonable (but practically unreachable) upper bound on it. The non-metadata buffer allocations, both mapped and unmapped, is accounted against maxbufspace, as before. Effectively, this means that the maxbufspace is forced on mapped and unmapped buffers separately. The pre-patch bufspace limiting code did not worked, because buffer_map fragmentation does not allow the limit to be reached. By Jeff Roberson request, the getnewbuf() function was split into smaller single-purpose functions. Sponsored by: The FreeBSD Foundation Discussed with: jeff (previous version) Tested by: pho, scottl (previous version), jhb, bf MFC after: 2 weeks
2013-03-19 14:13:12 +00:00
bip->bio_done = g_vfs_done;
bip->bio_caller2 = bp;
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
g_io_request(bip, cp);
}
Make various random things static
2005-02-10 12:10:35 +00:00
static void
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
g_vfs_orphan(struct g_consumer *cp)
{
Implement g_vfs_orphan(). Without it, the filesystem never closes the device, which means refcount on periph drivers never drops, which means cam_sim_free() never returns, which results in umass sleeping there ad infinitum. Submitted by: pjd Reviewed by: scottl, pjd Approved by: rwatson (mentor) Sponsored by: FreeBSD Foundation
2008-12-16 17:04:52 +00:00
struct g_geom *gp;
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
struct g_vfs_softc *sc;
int destroy;
Implement g_vfs_orphan(). Without it, the filesystem never closes the device, which means refcount on periph drivers never drops, which means cam_sim_free() never returns, which results in umass sleeping there ad infinitum. Submitted by: pjd Reviewed by: scottl, pjd Approved by: rwatson (mentor) Sponsored by: FreeBSD Foundation
2008-12-16 17:04:52 +00:00
g_topology_assert();
gp = cp->geom;
g_trace(G_T_TOPOLOGY, "g_vfs_orphan(%p(%s))", cp, gp->name);
Close race between geom destruction on g_vfs_close() when softc destroyed and g_vfs_orphan() call that tries to access softc, intruced at r227015. PR: kern/162997
2011-12-02 17:09:48 +00:00
sc = gp->softc;
if (sc == NULL)
return;
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
mtx_lock(&sc->sc_mtx);
sc->sc_orphaned = 1;
Partially revert r238886 in part of GEOM_VFS spoiling. This change triggered interesting foot shooting condition in GEOM when RW access to root partition by fsck spoils VFS geom there, which has it opened RO at the same time. Seems spoiling concept needs some rework.
2012-07-29 20:04:09 +00:00
destroy = (sc->sc_active == 0);
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
mtx_unlock(&sc->sc_mtx);
if (destroy)
g_vfs_destroy(cp, 0);
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
/*
Prevent a panic that happens on SMP machines when removing a disk with many writes queued up. Reviewed by: phk, scottl Approved by: rwatson (mentor) Sponsored by: FreeBSD Foundation
2009-01-11 13:51:04 +00:00
* Do not destroy the geom. Filesystem will do that during unmount.
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
*/
}
int
g_vfs_open(struct vnode *vp, struct g_consumer **cpp, const char *fsname, int wr)
{
struct g_geom *gp;
struct g_provider *pp;
struct g_consumer *cp;
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
struct g_vfs_softc *sc;
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
struct bufobj *bo;
int error;
g_topology_assert();
*cpp = NULL;
g_vfs_open: allow only one mount per device vnode In other words, deny multiple read-only mounts of the same device. Shared read-only mounts should theoretically be possible, but, unfortunately, can not be implemented correctly using current buffer cache code/interface and results in an eventual system crash. Also, using nullfs seems to be a more efficient way to achieve the same goal. This gets us back to where we were before GEOM and where other BSDs are. Submitted by: pjd (idea for checking for shared mounting) Discussed with: phk, pjd Silence from: fs@, geom@ MFC after: 2 weeks
2010-04-03 08:53:53 +00:00
bo = &vp->v_bufobj;
if (bo->bo_private != vp)
return (EBUSY);
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
pp = g_dev_getprovider(vp->v_rdev);
if (pp == NULL)
return (ENOENT);
gp = g_new_geomf(&g_vfs_class, "%s.%s", fsname, pp->name);
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
sc = g_malloc(sizeof(*sc), M_WAITOK | M_ZERO);
mtx_init(&sc->sc_mtx, "g_vfs", NULL, MTX_DEF);
sc->sc_bo = bo;
gp->softc = sc;
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
cp = g_new_consumer(gp);
g_attach(cp, pp);
Allow disk partitions associated with UFS read-only mounted filesystems to be opened for writing. This functionality used to be special-cased for just the root filesystem, but with this change is now available for all UFS filesystems. This change is needed for journaled soft updates recovery. Discussed with: Jeff Roberson
2011-07-10 00:41:31 +00:00
error = g_access(cp, 1, wr, wr);
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
if (error) {
g_wither_geom(gp, ENXIO);
return (error);
}
Create a correctly sized vnode objects for disk devices.
2005-01-24 22:41:21 +00:00
vnode_create_vobject(vp, pp->mediasize, curthread);
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
*cpp = cp;
g_vfs_open: allow only one mount per device vnode In other words, deny multiple read-only mounts of the same device. Shared read-only mounts should theoretically be possible, but, unfortunately, can not be implemented correctly using current buffer cache code/interface and results in an eventual system crash. Also, using nullfs seems to be a more efficient way to achieve the same goal. This gets us back to where we were before GEOM and where other BSDs are. Submitted by: pjd (idea for checking for shared mounting) Discussed with: phk, pjd Silence from: fs@, geom@ MFC after: 2 weeks
2010-04-03 08:53:53 +00:00
cp->private = vp;
Merge GEOM direct dispatch changes from the projects/camlock branch. When safety requirements are met, it allows to avoid passing I/O requests to GEOM g_up/g_down thread, executing them directly in the caller context. That allows to avoid CPU bottlenecks in g_up/g_down threads, plus avoid several context switches per I/O. The defined now safety requirements are: - caller should not hold any locks and should be reenterable; - callee should not depend on GEOM dual-threaded concurency semantics; - on the way down, if request is unmapped while callee doesn't support it, the context should be sleepable; - kernel thread stack usage should be below 50%. To keep compatibility with GEOM classes not meeting above requirements new provider and consumer flags added: - G_CF_DIRECT_SEND -- consumer code meets caller requirements (request); - G_CF_DIRECT_RECEIVE -- consumer code meets callee requirements (done); - G_PF_DIRECT_SEND -- provider code meets caller requirements (done); - G_PF_DIRECT_RECEIVE -- provider code meets callee requirements (request). Capable GEOM class can set them, allowing direct dispatch in cases where it is safe. If any of requirements are not met, request is queued to g_up or g_down thread same as before. Such GEOM classes were reviewed and updated to support direct dispatch: CONCAT, DEV, DISK, GATE, MD, MIRROR, MULTIPATH, NOP, PART, RAID, STRIPE, VFS, ZERO, ZFS::VDEV, ZFS::ZVOL, all classes based on g_slice KPI (LABEL, MAP, FLASHMAP, etc). To declare direct completion capability disk(9) KPI got new flag equivalent to G_PF_DIRECT_SEND -- DISKFLAG_DIRECT_COMPLETION. da(4) and ada(4) disk drivers got it set now thanks to earlier CAM locking work. This change more then twice increases peak block storage performance on systems with manu CPUs, together with earlier CAM locking changes reaching more then 1 million IOPS (512 byte raw reads from 16 SATA SSDs on 4 HBAs to 256 user-level threads). Sponsored by: iXsystems, Inc. MFC after: 2 months
2013-10-22 08:22:19 +00:00
cp->flags |= G_CF_DIRECT_SEND | G_CF_DIRECT_RECEIVE;
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
bo->bo_ops = g_vfs_bufops;
bo->bo_private = cp;
bo_bsize: revert r205860 and take an alternative approch in getblk In r205860 I missed the fact that there is code that strongly assumes that devvp bo_bsize is equal to underlying provider's sectorsize. In those places it is hard to obtain the sectorsize in an alternative way if devvp bo_bsize is set to something else. So, I am reverting bo_bsize assigment in g_vfs_open. Instead, in getblk I use DEV_BSIZE block size for b_offset calculation if vp is a disk vp as reported by vn_isdisk. This should coinside with vp being a devvp. Reported by: Mykola Dzham <i@levsha.me> Tested by: Mykola Dzham <i@levsha.me> Pointyhat to: avg MFC after: 2 weeks X-ToDo: convert bread(devvp) in all fs to use bo_bsize-d blocks
2010-04-02 15:12:31 +00:00
bo->bo_bsize = pp->sectorsize;
Add GEOM class "VFS" for filesystems and other buffer cache users of GEOM devices. There is nothing magic about this, it just gives a bufobj interface to GEOM.
2004-10-29 09:56:56 +00:00
return (error);
}
Introduce and use g_vfs_close().
2005-01-25 15:52:04 +00:00
void
Remove the struct thread unuseful argument from bufobj interface. In particular following functions KPI results modified: - bufobj_invalbuf() - bufsync() and BO_SYNC() "virtual method" of the buffer objects set. Main consumers of bufobj functions are affected by this change too and, in particular, functions which changed their KPI are: - vinvalbuf() - g_vfs_close() Due to the KPI breakage, __FreeBSD_version will be bumped in a later commit. As a side note, please consider just temporary the 'curthread' argument passing to VOP_SYNC() (in bufsync()) as it will be axed out ASAP Reviewed by: kib Tested by: Giovanni Trematerra <giovanni dot trematerra at gmail dot com>
2008-10-10 21:23:50 +00:00
g_vfs_close(struct g_consumer *cp)
Introduce and use g_vfs_close().
2005-01-25 15:52:04 +00:00
{
struct g_geom *gp;
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
struct g_vfs_softc *sc;
Introduce and use g_vfs_close().
2005-01-25 15:52:04 +00:00
g_topology_assert();
gp = cp->geom;
Add mutex and two flags to make orphan() call properly asynchronous: - delay consumer closing and detaching on orphan() until all I/Os complete; - prevent new I/Os submission after orphan() called. Previous implementation could destroy consumers still having active requests and worked only because of global workaround made on GEOM level.
2011-11-02 09:24:59 +00:00
sc = gp->softc;
bufobj_invalbuf(sc->sc_bo, V_SAVE, 0, 0);
sc->sc_bo->bo_private = cp->private;
gp->softc = NULL;
mtx_destroy(&sc->sc_mtx);
if (!sc->sc_orphaned || cp->provider == NULL)
g_wither_geom_close(gp, ENXIO);
g_free(sc);
Introduce and use g_vfs_close().
2005-01-25 15:52:04 +00:00
}
Reference in New Issue Copy Permalink