Commit Graph

985 Commits

Author SHA1 Message Date
Xin LI
26f96d922b Fix a regression introduced in r274337 (large block support)
In dsl_dataset_hold_obj() we used zap_contains(.., DS_FIELD_LARGE_BLOCKS)
to determine whether the extensible (zapifyed) dataset have large blocks.
The code expects the result be either 0 (found) or ENOENT (not found),
however reused the variable 'err' which later code expects to be 0.

Fix this by adopting similar code construct that is used later for
DS_FIELD_BOOKMARK_NAMES, which uses a temporary variable zaperr to catch
errors from zap_* rountines.

Reported by:	Peter J. Creath (on FreeNAS; FreeNAS bug #6848)
Illumos issue:	5393 spurious failures from dsl_dataset_hold_obj()
Reviewed by:	mahrens
Sponsored by:	iXsystems, Inc.
X-MFC with:	r274337
2014-12-05 18:29:01 +00:00
Alexander Motin
ef8daf3fed Add GET LBA STATUS command support to CTL.
It is implemented for LUNs backed by ZVOLs in "dev" mode and files.
GEOM has no such API, so for LUNs backed by raw devices all LBAs will
be reported as mapped/unknown.

MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
2014-12-04 11:34:19 +00:00
Andriy Gapon
782c06dfc8 zfs_putpages: actually update mtime and ctime
Reported by:	Paul Koch <paul.koch@akips.com>
Tested by:	Paul Koch <paul.koch@akips.com>
MFC after:	2 weeks
2014-12-02 11:44:56 +00:00
Xin LI
d10e52d627 Revert r273060 per discussion with avg@ as we need to make L2ARC
aware of 4K devices and this one is not the right fix anyway.
2014-11-26 02:20:25 +00:00
Dimitry Andric
c577699e4b Fix the following -Werror warning from clang 3.5.0, while building cddl/lib/libctf:
In file included from cddl/contrib/opensolaris/common/ctf/ctf_create.c:31:
In file included from sys/cddl/contrib/opensolaris/uts/common/sys/sysmacros.h:34:
sys/cddl/contrib/opensolaris/uts/common/sys/isa_defs.h:334:9: warning: '_ILP32' macro redefined [-Wmacro-redefined]
#define _ILP32
        ^
<built-in>:26:9: note: previous definition is here
#define _ILP32 1
        ^
1 warning generated.

This is because clang 3.5.0 started predefining _ILP32 and __ILP32__ for
the i386 arch.  (Earlier versions already predefined _LP64 and __LP64__
for the x86_64 arch.)

Reviewed by:	emaste, avg, smh, delphij, markj
Differential Revision: https://reviews.freebsd.org/D1187
2014-11-19 07:44:21 +00:00
Xin LI
8c3d6a4ab2 Make vfs.zfs.max_recordsize read-write at runtime.
MFC after:	2 weeks
2014-11-18 22:35:19 +00:00
Xin LI
8efcd876ca Add a tunable for spa_slop_shift which controls how much space we
would reserve by default.  Tuning is not recommended.

MFC after:	2 weeks
2014-11-18 18:52:38 +00:00
Xin LI
18144ab1a3 Allow tuning zfs_max_recordsize via loader tunable. Tuning is NOT
recommended.

Requested by:	Slawa Olhovchenkov <slw zxy spb ru>
MFC after:	2 weeks
2014-11-18 18:40:01 +00:00
Andriy Gapon
2c51c83bc8 l2arc: restore correct rounding up of asize of compressed data
This rounding up was lost in a mismerge of illumos code.
See r268075 MFV r267565.
After that commit zio_compress_data() no longer performs any compressed
size adjustment, so it needs to be done externally.  On FreeBSD we round
up the size using vdev_ashift rather than SPA_MINBLOCKSIZE so that 4KB
devices are properly supported.

Additionally, zero out the buffer tail only if compression succeeds.
The compression is considered successful if the size of compressed
data after rounding up to account for the vdev ashift is less than the
original data size.  It does not make sense to have the data compressed
if all the savings are lost to rounding up.
With the new zio_compress_data() it could have been possible that the
rounded compressed size would be greater than the original size and thus
we could zero beyond the allocated buffer if the zeroing code was kept
at the original place.

Discussed with:	delphij, gibbs
MFC after:	2 weeks
X-MFC with:	r274627
2014-11-17 14:45:42 +00:00
Andriy Gapon
0908b20b7e Revert r269093 which introduced physical zio alignment transform
Size of physical ZIOs must never be implicitly adjusted, it's
a responsibility of a caller to make sure that such a ZIO has proper offset
and size.

Discussed with:	delphij, gibbs
MFC after:	2 weeks
2014-11-17 14:16:02 +00:00
Steven Hartland
a559adfbce Disable TRIM on file backed ZFS vdevs and fix TRIM on init
After r265152 TRIM requests are ZIO_TYPE_FREE instead of ZIO_TYPE_IOCTL
this meant file backed vdevs to attempted to process the ZIO as a write
causing a panic.

We now disable TRIM on file backed vdevs and ASSERT the ZIO types supported
by each vdev type to ensure we explicity support the ZIO type being
processed.

Also ensure that TRIM on init is not procesed for devices which declare they
didn't support TRIM via vdev_notrim.

PR:		195061, 194976, 191573
Sponsored by:	Multiplay
2014-11-17 11:32:10 +00:00
Xin LI
8bcd603968 MFV r274273:
ZFS large block support.

Please note that booting from datasets that have recordsize greater
than 128KB is not supported (but it's Okay to enable the feature on
the pool).  This *may* remain unchanged because of memory constraint.

Limited safety belt is provided for mounted root filesystem but use
caution is advised.

Illumos issue:
    5027 zfs large block support

MFC after:	1 month
2014-11-10 08:20:21 +00:00
Xin LI
42350b6bde MFV r274272 and diff reduction with upstream.
Illumos issue:
    5244 zio pipeline callers should explicitly invoke next stage

Tested with:	ztest plus ZFS over GELI configuration
MFC after:	1 month
2014-11-09 07:37:00 +00:00
Xin LI
81f1255e58 MFV r274271:
Improve zdb -b performance:

 - Reduce gethrtime() call to 1/100th of blkptr's;
 - Skip manipulating the size-ordered tree;
 - Issue more (10, previously 3) async reads;
 - Use lighter weight testing in traverse_visitbp();

Illumos issue:
    5243 zdb -b could be much faster

MFC after:	2 weeks
2014-11-08 07:30:40 +00:00
Andriy Gapon
2fd3cc0cb2 fix l2arc compression buffers leak
We have observed that arc_release() can be called concurrently with a
l2arc in-flight write.
Also, we have observed that arc_hdr_destroy() can be called from
arc_write_done() for a zio with ZIO_FLAG_IO_REWRITE flag in similar
circumstances.

Previously the l2arc headers would be freed while leaking their
associated compression buffers.  Now the buffers are placed on
l2arc_free_on_write list for delayed freeing.  This is similar to what
was already done to arc buffers that were supposed to be freed
concurrently with in-flight writes of those buffers.

In addition to fixing the discovered leaks this change also adds some
protective code to assert that a compression buffer associated with a
l2arc header is never leaked.

A new kstat l2_cdata_free_on_write is added.  It keeps a count of
delayed compression buffer frees which previously would have been leaks.

Tested by:	Vitalij Satanivskij <satan@ukr.net> et al
Requested by:	many
MFC after:	2 weeks
Sponsored by:	HybridCluster / ClusterHQ
2014-11-06 11:08:02 +00:00
Alexander Motin
c3e7ba3e6d Add to CTL support for logical block provisioning threshold notifications.
For ZVOL-backed LUNs this allows to inform initiators if storage's used or
available spaces get above/below the configured thresholds.

MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
2014-11-06 00:48:36 +00:00
Josh Paetzel
14127f5b21 This change addresses 4 bugs in ZFS exposed by Richard Kojedzinszky's
crash.sh script attached to FreeNAS bug 4109:
https://bugs.freenas.org/issues/4109

Three are in the snapshot layer:
a) AVG explains in his notes: https://wiki.freebsd.org/AvgVfsSolarisVsFreeBSD

"VOP_INACTIVE must not do any destructive actions to a vnode
and its filesystem node, nor invalidate them in any way."
gfs_vop_inactive and zfsctl_snapshot_inactive did just that. In
OpenSolaris VOP_INACTIVE is much closer to FreeBSD's VOP_RECLAIM.
Rename & move them to gfs_vop_reclaim and zfsctl_snapshot_reclaim
and merge in the requisite vnode_destroy from zfsctl_common_reclaim.

b) gfs_lookup_dot and various zfsctl functions do not honor the
FreeBSD VFS convention of only locking from the root downward. When
looking up ".." the convention is to drop the current leaf vnode lock before
acquiring the directory vnode and then subsequently re-acquiring the lock on the
leaf vnode. This fixes that in all the places that our exercised by crash.sh.

c) The snapshot may already be unmounted when the directory vnode is reclaimed.
Check for this case and return.

One in the common layer:
d) Callers of traverse expect the reference to the vnode passed in to be
maintained. Don't release it.

This last one may be an unclear contract. There may in fact be some callers that
do expect the reference to be dropped on success in addition to callers that
expect it to be released. In this case a further audit of the callers is needed
and a consensus on the correct behavior.

PR:	184677
Submitted by:	kmacy
Reviewed by:	delphij, will, avg
MFC after:	2 weeks
Sponsored by:	iXsystems
2014-10-25 17:42:44 +00:00
Hans Petter Selasky
f0188618f2 Fix multiple incorrect SYSCTL arguments in the kernel:
- Wrong integer type was specified.

- Wrong or missing "access" specifier. The "access" specifier
sometimes included the SYSCTL type, which it should not, except for
procedural SYSCTL nodes.

- Logical OR where binary OR was expected.

- Properly assert the "access" argument passed to all SYSCTL macros,
using the CTASSERT macro. This applies to both static- and dynamically
created SYSCTLs.

- Properly assert the the data type for both static and dynamic
SYSCTLs. In the case of static SYSCTLs we only assert that the data
pointed to by the SYSCTL data pointer has the correct size, hence
there is no easy way to assert types in the C language outside a
C-function.

- Rewrote some code which doesn't pass a constant "access" specifier
when creating dynamic SYSCTL nodes, which is now a requirement.

- Updated "EXAMPLES" section in SYSCTL manual page.

MFC after:	3 days
Sponsored by:	Mellanox Technologies
2014-10-21 07:31:21 +00:00
Xin LI
78701de4b7 Add tunable vfs.zfs.space_map_blksz for space map's maximum block size.
MFC after:	2 weeks
2014-10-18 22:11:10 +00:00
Davide Italiano
2be111bf7d Follow up to r225617. In order to maximize the re-usability of kernel code
in userland rename in-kernel getenv()/setenv() to kern_setenv()/kern_getenv().
This fixes a namespace collision with libc symbols.

Submitted by:   kmacy
Tested by:      make universe
2014-10-16 18:04:43 +00:00
Steven Hartland
ca6505b818 Prevent ZFS leaking pool free space
When processing async destroys ZFS would leak space every txg timeout
(5 seconds by default), if no writes occurred, until the pool is totally
full. At this point it would be unfixable without a pool recreation.

In addition if the machine was rebooted with the pool in this situation
would fail to import on boot, hanging indefinitely, as the import process
requires the ability to write data to the pool. Any attempts to query the
pool status during the hung import would not return as the import holds
the pool lock.

The only way to import such a pool would be to specify -o readonly=on
to the zpool import.

zdb -bb <pool> can be used to check for "deferred free" size which is where
this lost space will be counted.

MFC after:	3 days
Sponsored by:	Multiplay
2014-10-16 02:23:27 +00:00
Xin LI
ba6e85e0cf Use write_psize instead of write_asize when doing vdev_space_update.
Without this change the accounting of L2ARC usage would be wrong and
give 16EB free space because the number became negative and overflows.

Obtained from:	FreeNAS (issue #6239)
MFC after:	2 weeks
2014-10-13 20:39:51 +00:00
Xin LI
a4f5b8db9f Add a tunable for arc_shrink_shift (vfs.zfs.arc_shrink_shift) that
controls how much fraction, 1/2^arc_shrink_shift, should be reclaimed
when there is memory pressure.

Submitted by:	Richard Kojedzinszky <krichy at tvnetwork.hu>
MFC after:	2 weeks
2014-10-13 05:34:10 +00:00
Xin LI
eba15cf463 MFV r272804:
Refactor the code and stop restore_object from creating two transactions.

Illumos issue:
    3693 restore_object uses at least two transactions to restore an object

MFC after:	2 weeks
2014-10-09 07:52:51 +00:00
Xin LI
ce44f14b41 MFV r272803:
Illumos issue:
    5175 implement dmu_read_uio_dbuf() to improve cached read performance

MFC after:	2 weeks
2014-10-09 07:18:40 +00:00
Andriy Gapon
c3d1d2e104 l2arc_write_buffers: reduce headroom value
FreeBSD has ARC_BUFC_NUMMETADATALISTS metadata lists and ARC_BUFC_NUMDATALISTS
data lists (currently both are 16) while illumos has just a single list
of each kind.

headroom determines how much data is scanned on a single list
during each run of the l2arc feed thread.
Because FreeBSD has more lists we proportionally decrease the limit.

Reviewed by:	Brendan Gregg (earlier version)
MFC after:	2 weeks
Sponsored by:	HybridCluster
2014-10-07 16:08:21 +00:00
Andriy Gapon
9f96723ec5 revert r272702: wrong (earlier) change was committed 2014-10-07 16:06:10 +00:00
Andriy Gapon
4c3b02bfce reduce L2ARC_WRITE_SIZE on FreeBSD
FreeBSD has ARC_BUFC_NUMMETADATALISTS metadata lists and ARC_BUFC_NUMDATALISTS
data lists (currently both are 16) while illumos has just a single list
of each kind.

L2ARC_WRITE_SIZE determines the default value of l2arc_write_max which
defines limits on how much data is scanned and written to a cache device
during each run of the l2arc feed thread.  The limits are applied on the
per buffer list basis.
Because FreeBSD has more lists we proportionally reduce the limits.

Reviewed by:	Brendan Gregg (earlier version)
MFC after:	2 weeks
Sponsored by:	HybridCluster
2014-10-07 14:30:24 +00:00
Xin LI
1b5bcb8425 MFV r272591:
Use loaned ARC buffer for zfs receive to avoid copy.

Illumos issue:
    5162 zfs recv should use loaned arc buffer to avoid copy

MFC after:	2 weeks
2014-10-06 07:29:17 +00:00
Xin LI
8fb26f5aef MFV r272585:
Split the godfather zio into CPU number's to reduce lock
contention.

Illumos issue:
    5176 lock contention on godfather zio

MFC after:	2 weeks
2014-10-06 07:03:17 +00:00
Xin LI
dcb20006f0 MFV r272501:
Illumos issue:
    5177 remove dead code from dsl_scan.c

MFC after:	2 weeks
2014-10-06 05:46:51 +00:00
Xin LI
00769ce74d MFV r272500:
Don't inherit flags other than DS_FLAG_CI_DATASET and DS_FLAG_INCONSISTENT
when cloning.  This prevents DS_FLAG_DEFER_DESTROY being inherited from a
clone that is marked for deferred destroy, which causes snapshots of the
clone being destroyed when getting a hold or clone.

Illumos issue:
    5150 zfs clone of a defer_destroy snapshot causes strangeness

MFC after:	1 week
2014-10-06 05:42:20 +00:00
Xin LI
4bb264ae15 Don't make nested definition for range_seg_cache.
Reported by:	ian
MFC after:	1 week
X-MFC-With:	r272506
2014-10-04 15:42:52 +00:00
Xin LI
4750c382a9 MFV r272499:
Illumos issue:
    5174 add sdt probe for blocked read in dbuf_read()

MFC after:	2 weeks
2014-10-04 08:55:08 +00:00
Xin LI
eb0b70068c Add a new sysctl, vfs.zfs.vol.unmap_enabled, which allows the system
administrator to toggle whether ZFS should ignore UNMAP requests.

Illumos issue:
    5149 zvols need a way to ignore DKIOCFREE

MFC after:	2 weeks
2014-10-04 08:51:57 +00:00
Xin LI
2d36d67c72 Diff reduction with upstream. The code change is not really applicable
to FreeBSD.

Illumos issue:
    5148 zvol's DKIOCFREE holds zfsdev_state_lock too long

MFC after:	1 month
2014-10-04 08:41:23 +00:00
Xin LI
523b4c7fdf MFV r272496:
Add tunable for number of metaslabs per vdev
(vfs.zfs.vdev.metaslabs_per_vdev).  The default remains
at 200.

Illumos issue:
    5161 add tunable for number of metaslabs per vdev

MFC after:	2 weeks
2014-10-04 08:29:48 +00:00
Xin LI
a8d7512709 MFV r272495:
In arc_kmem_reap_now(), reap range_seg_cache too to reclaim memory in
response of memory pressure.

Illumos issue:
    5163 arc should reap range_seg_cache

MFC after:	1 week
2014-10-04 08:14:10 +00:00
Xin LI
8c20e2ff11 MFV r272494:
Make space_map_truncate() always do space_map_reallocate().  Without
this, setting space_map_max_blksz would cause panic for existing pool,
as dmu_objset_set_blocksize would fail if the object have multiple blocks.

Illumos issues:
   5164 space_map_max_blksz causes panic, does not work
   5165 zdb fails assertion when run on pool with recently-enabled
	spacemap_histogram feature

MFC after:	2 weeks
2014-10-04 08:05:39 +00:00
Steven Hartland
14a0d74ea8 Refactor ZFS ARC reclaim checks and limits
Remove previously added kmem methods in favour of defines which
allow diff minimisation between upstream code base.

Rebalance ARC free target to be vm_pageout_wakeup_thresh by default
which eliminates issue where ARC gets minimised instead of balancing
with VM pageout. The restores the target point prior to r270759.

Bring in missing upstream only changes which move unused code to
further eliminate code differences.

Add additional DTRACE probe to aid monitoring of ARC behaviour.

Enable upstream i386 code paths on platforms which don't define
UMA_MD_SMALL_ALLOC.

Fix mixture of byte an page values in arc_memory_throttle i386 code
path value assignment of available_memory.

PR:		187594
Review:		D702
Reviewed by:	avg
MFC after:	1 week
X-MFC-With:	r270759 & r270861
Sponsored by:	Multiplay
2014-10-03 20:34:55 +00:00
Steven Hartland
99140218aa Fix various issues with zvols
When performing snapshot renames we could deadlock due to the locking
in zvol_rename_minors. In order to avoid this use the same workaround
as zvol_open in zvol_rename_minors.

Add missing zvol_rename_minors to dsl_dataset_promote_sync.

Protect against invalid index into zv_name in zvol_remove_minors.

Replace zvol_remove_minor calls with zvol_remove_minors to ensure
any potential children are also renamed.

Don't fail zvol_create_minors if zvol_create_minor returns EEXIST.

Restore the valid pool check in zfs_ioc_destroy_snaps to ensure we
don't call zvol_remove_minors when zfs_unmount_snap fails.

PR:		193803
MFC after:	1 week
Sponsored by:	Multiplay
2014-10-03 14:49:48 +00:00
Marcelo Araujo
d8a5961f88 Fix failures and warnings reported by newpynfs20090424 test tool.
This fix addresses only issues with the pynfs reports, none of these
issues are know to create problems for extant real clients.

Submitted by:	Bart Hsiao <bart.hsiao@gmail.com>
Reworked by:	myself
Reviewed by:	rmacklem
Approved by:	rmacklem
Sponsored by:	QNAP Systems Inc.
2014-10-03 02:24:41 +00:00
Will Andrews
fbce0221eb zfsvfs_create(): Refuse to mount datasets whose names are too long.
This is checked for in the zfs_snapshot_004_neg STF/ATF test (currently
still in projects/zfsd rather than head).

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:
- zfsvfs_create(): Check whether the objset name fits into
  statfs.f_mntfromname, and return ENAMETOOLONG if not.  Although
  the filesystem can be unmounted via the umount(8) command, any
  interface that relies on iterating on statfs (e.g. libzfs) will
  fail to find the filesystem by its objset name, and thus assume
  it's not mounted.  This causes "zfs unmount", "zfs destroy",
  etc. to fail on these filesystems, whether or not -f is passed.

MFC after:	1 month
Sponsored by:	Spectra Logic
MFSpectraBSD:	974872 on 2013/08/09
2014-10-01 14:12:02 +00:00
Xin LI
0b66c7c514 Fix a mismerge in r260183 which prevents snapshot zvol devices being
removed and re-instate the fix in r242862.

Reported by:	Leon Dang <ldang nahannisys com>, smh
MFC after:	3 days
2014-09-30 18:50:45 +00:00
Will Andrews
91dda985cc Remove debug.zfs_flags in favor of the new vfs.zfs.debug_flags.
Replace TUNABLE_INT with CTLFLAG_RWTUN.

Submitted by:	avg (debug.zfs_flags removal), smh (TUNABLE_INT replacement)
2014-09-18 18:46:38 +00:00
Will Andrews
f8c2f66a6c Enable ZFS debug flags to be modified via vfs.zfs.debug_flags.
This is primarily only of interest to ZFS developers, but it makes it
easier to get additional debugging.

Submitted by:	gibbs
MFC after:	1 month
Sponsored by:	Spectra Logic
MFSpectraBSD:	517074 on 2011/12/15 (by will), 662343 on 2013/03/20 (by gibbs)
2014-09-18 16:55:41 +00:00
Will Andrews
cf0a1157d7 Reorder sysctls for spa.c global tunables; add sysctl for ccw_retry_interval.
MFC after:	1 month
Sponsored by:	Spectra Logic
2014-09-18 16:38:03 +00:00
Will Andrews
cf7a096e72 bpobj_iterate_impl(): Close a refcount leak iterating on a sublist.
If bpobj_space() returned non-zero here, the sublist would have been
left open, along with the bonus buffer hold it requires.  This call
does not invoke any calls to bpobj_close() itself.

This bug doesn't have any known vector, but was found on inspection.

MFC after:	1 week
Sponsored by:	Spectra Logic
Affects:	All ZFS versions starting 21 May 2010 (illumos cde58dbc)
MFSpectraBSD:	r1050998 on 2014/03/26
2014-09-18 15:37:53 +00:00
Steven Hartland
d1d469e22b Remove unused ZFS ARC functions
* arc_data_buf_alloc
* arc_data_buf_free

MFC after:	1 week
Sponsored by:	Multiplay
2014-09-18 10:46:51 +00:00
Steven Hartland
a889b18c52 Added missing ZFS sysctls
* vfs.zfs.vdev.async_write_active_min_dirty_percent
* vfs.zfs.vdev.async_write_active_max_dirty_percent

Added validation of min / max for ZFS sysctl
* vfs.zfs.dirty_data_max_percent

MFC after:	3 days
2014-09-14 12:23:00 +00:00
Xin LI
f9290bc2c9 MFV r271518:
Correctly report hole at end of file.

When asked to find a hole, the DMU sees that there are no holes in the
object, and returns ESRCH.  The ZPL interprets this as "no holes before
the end of the file", and therefore inserts the "virtual hole" at the
end of the file.  Because DMU and ZPL have different ideas of where the
end of an object/file is, we will end up returning the end of file,
which is generally larger, instead of returning the end of object.

The fix is to handle the "virtual hole" in the DMU. If no hole is found,
the DMU will return a hole at the end of the file, rather than an error.

Illumos issue:
    5139 SEEK_HOLE failed to report a hole at end of file

MFC after:	1 week
2014-09-13 17:48:44 +00:00
Xin LI
dc147754b7 MFV r271517:
In zil_claim, don't issue warning if we get EBUSY (inconsistent) when
opening an objset, instead, ignore it silently.

Illumos issue:

    5140 message about "%recv could not be opened" is printed when booting after crash

MFC after:	1 week
2014-09-13 17:36:34 +00:00
Xin LI
be1b14a063 MFV r271515:
Add a new tunable/sysctl, vfs.zfs.free_max_blocks, which can be used to
limit how many blocks can be free'ed before a new transaction group is
created.  The default is no limit (infinite), but we should probably have
a lower default, e.g. 100,000.

With this limit, we can guard against the case where ZFS could run out of
memory when destroying large numbers of blocks in a single transaction
group, as the entire DDT needs to be brought into memory.

Illumos issue:
    5138 add tunable for maximum number of blocks freed in one txg

MFC after:	2 weeks
2014-09-13 17:24:56 +00:00
Xin LI
ff0fc48bde MFV r271512:
Illumos issue:
    5136 fix write throttle comment in dsl_pool.c

MFC after:	2 weeks
2014-09-13 16:51:23 +00:00
Xin LI
263f396e2b MFV r271510:
Enforce 4K as smallest indirect block size (previously the smallest
indirect block size was 1K but that was never used).

This makes some space estimates more accurate and uses less memory
for some data structures.

Illumos issue:
    5141 zfs minimum indirect block size is 4K

MFC after:	2 weeks
2014-09-13 16:26:14 +00:00
Steven Hartland
3cdd9138c3 Persist vdev_resilver_txg changes to avoid panic caused by validation
vs a vdev_resilver_txg value from a previous resilver.

MFC after:	1 week
2014-09-11 16:21:51 +00:00
Gleb Smirnoff
27ad26d8c7 Remove unused arguments for VOP_GETPAGES(), VOP_PUTPAGES(). 2014-09-10 12:36:41 +00:00
Alexander Motin
ee9534ed96 Make ZVOL writes in device mode support IO_SYNC flag.
MFC after:	1 month
2014-09-09 11:29:55 +00:00
Xin LI
817d804595 MFV r271223:
In dnode_sync(), do dnode_increase_indirection() before processing
the dn_next_nblkptr.

Illumos issue:
    5117 space map reallocation can cause corruption

MFC after:	3 days
2014-09-07 13:13:42 +00:00
Peter Wemm
d903c21a64 Move the restored #ifdef i386 test back inside the #ifdef _KERNEL block
where it originally was.
2014-08-31 09:05:02 +00:00
Steven Hartland
92ac3eb59f Ensure that ZFS ARC free memory checks include cached pages
Also restore kmem_used() check for i386 as it has KVA limits that the raw
page counts above don't consider

PR:		187594
Reviewed by:	peter
X-MFC-With: r270759
Review:	D700
Sponsored by:	Multiplay
2014-08-30 21:44:32 +00:00
Mateusz Guzik
6662ce5aab Add missing proctree locking to fill_kinfo_proc consumers.
This fixes r270444.

Pointy hat:	mjg
Reported by:	many
MFC after:	1 week
2014-08-30 03:10:55 +00:00
Steven Hartland
4d19f4ad1f Refactor ZFS ARC reclaim logic to be more VM cooperative
Prior to this change we triggered ARC reclaim when kmem usage passed 3/4
of the total available, as indicated by vmem_size(kmem_arena, VMEM_ALLOC).

This could lead large amounts of unused RAM e.g. on a 192GB machine with
ARC the only major RAM consumer, 40GB of RAM would remain unused.

The old method has also been seen to result in extreme RAM usage under
certain loads, causing poor performance and stalls.

We now trigger ARC reclaim when the number of free pages drops below the
value defined by the new sysctl vfs.zfs.arc_free_target, which defaults
to the value of vm.v_free_target.

Credit to Karl Denninger for the original patch on which this update was
based.

PR:		191510 and 187594
Tested by:	dteske
MFC after:	1 week
Relnotes:	yes
Sponsored by:	Multiplay
2014-08-28 19:50:08 +00:00
Xin LI
ec1b564650 Instead of using timestamp in the AVL, use the memory address when
comparing.

Illumos issue:
    5095 panic when adding a duplicate dbuf to dn_dbufs

MFC after:	3 days
2014-08-22 23:13:53 +00:00
Xin LI
fa4484104c MFV r270197:
Illumos issue:
    5066 remove support for non-ANSI compilation
    5068 Remove SCCSID() macro from <macros.h>

MFC after:	2 weeks
2014-08-22 22:13:36 +00:00
Xin LI
7c1db36b28 MFV r270196:
Illumos issue:
    5047 don't use atomic_*_nv if you discard the return value

MFC after:	2 weeks
2014-08-20 22:39:26 +00:00
Xin LI
249ddb42f6 MFC r270195:
Illumos issue:
    5045 use atomic_{inc,dec}_* instead of atomic_add_*

MFC after:	2 weeks
2014-08-20 21:44:48 +00:00
Xin LI
2bcc37f99c MFV r270193:
Illumos issues:
    5042 stop using deprecated atomic functions

MFC after:	2 weeks
2014-08-20 18:29:18 +00:00
Xin LI
60723bfe21 MFV r269542:
In vdev_get_stats, check that the vdev is not a hole before computing the
fragmentation.  This fixes a panic when removing log device.

Illumos issue:
    5049 panic when removing log device

Author:		Alex Reece <alex@delphix.com>
MFC after:	2 weeks
2014-08-05 00:07:21 +00:00
Mark Johnston
2661328745 Return 0 for the PPID of threads in process 0, as process 0 doesn't have a
parent process.

MFC after:	2 weeks
2014-08-04 19:02:30 +00:00
Xin LI
cd741a5e1d Revert r269404 and use cpu_ticks() for dbuf allocation.
Encode CPU's number by XOR'ing the CPU ID against the 64-bit cpu_ticks().

Reviewed by:	mav, gibbs
Differential Revision: https://phabric.freebsd.org/D521
MFC after:	2 weeks
2014-08-03 09:47:51 +00:00
Xin LI
1dcef10eac MFV r269427:
In dnode_children_t, use C99's "[]" idiom for declaring the variable
sized array dnc_children at the end of the structure.

This prevents the compiler from mistakenly optimizing away accesses
beyond the array's defined size.

Illumos issue:
    5038 Remove "old-style" flexible array usage in ZFS.
    Author: Justin T. Gibbs <justing@spectralogic.com>

MFC after:	2 weeks
2014-08-02 08:34:22 +00:00
Steven Hartland
6a369c018c Don't return ZIO_PIPELINE_CONTINUE from vdev_op_io_start methods
This prevents recursion of vdev_queue_io_done as per r265321 but
using a different method as recommended on the openzfs list.

We now use zio_interrupt(zio) and return ZIO_PIPELINE_STOP instead
of returning ZIO_PIPELINE_CONTINUE from vdev_*_io_start methods.

zio_vdev_io_start now ASSERTS the that vdev_op_io_start returns
ZIO_PIPELINE_STOP to ensure future changes don't reintroduce
ZIO_PIPELINE_CONTINUE returns.

Cleanup flow in vdev_geom_io_start while I'm here.

Also fix some cases not using SET_ERROR(..)

MFC after:	2 weeks
X-MFC-With:	r265321
2014-08-01 23:16:48 +00:00
Rui Paulo
d18aa577d5 Copy strtolctype.h to sys/cddl/contrib/opensolaris/common/util to keep
the kernel self-contained.

Requested by:	jhb
2014-07-31 08:07:23 +00:00
Xin LI
9b046b421f MFV r269224:
Increase default ARC buf_hash_table size.  When typical block size is small,
the hash table could be too small, which would lead to long hash chains and
limit performance for cached reads.

A new loader tunable, vfs.zfs.arc_average_blocksize, have been added which
allows users to override the default assumption of average (typical) block
size.  Old default was 65536 (64 KiB) and new default is 8192 (8 KiB).

Illumos issue:
    5034 ARC's buf_hash_table is too small

MFC after:	2 weeks
2014-07-29 09:36:48 +00:00
Xin LI
a3cbca537e MFV r269223:
Change dn->dn_dbufs from linked list to AVL tree.

Illumos issues:
  4873 zvol unmap calls can take a very long time for larger datasets

MFC after:	2 weeks
2014-07-29 08:42:22 +00:00
Xin LI
343c95a24e Reschedule the 'deadman' callout after handling, this makes our
code behave more like it is on Solaris.

Reported by:	avg
Reviewed by:	avg, mav (but bugs are mine)

Differential Revision: https://phabric.freebsd.org/D457
2014-07-29 06:57:13 +00:00
Konstantin Belousov
fe0e9a63e0 Initialize zfs vnode v_hash when the vnode is allocated, instead of
postponing it to zfs_vget().  zfs_root() returned vnode with the
default value of v_hash, which caused inconsistent v_hash value when
root vnode was obtained from zfs_vget().

Nullfs allocated two upper vnodes for the root zfs vnode due to
different hashes, causing consistency problems.

Reported and tested by:	Harald Schmalzbauer <h.schmalzbauer@omnilan.de>
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2014-07-28 14:24:18 +00:00
Xin LI
50b74c6ef1 Add two sysctls for newly added tunables.
MFC after:	2 weeks
2014-07-26 19:07:08 +00:00
Xin LI
7e37b1e609 MFV r269010:
Import Illumos changes to address the following Illumos issues:
  4976 zfs should only avoid writing to a failing non-redundant
       top-level vdev
  4978 ztest fails in get_metaslab_refcount()
  4979 extend free space histogram to device and pool
  4980 metaslabs should have a fragmentation metric
  4981 remove fragmented ops vector from block allocator
  4982 space_map object should proactively upgrade when feature
       is enabled
  4984 device selection should use fragmentation metric

MFC after:	2 weeks
2014-07-26 10:20:48 +00:00
Alexander Motin
1bc04f6a8c Make sysctls under vfs.zfs.zfetch writeable.
I don't see any reason for them to be read-only, while tuning them without
reboot is much more convenient for experiments.

MFC after:	2 weeks
2014-07-26 09:09:14 +00:00
Xin LI
0aa4ce9b7d Transform the I/O when vdev_physical_ashift is greater than
SPA_MINBLOCKSHIFT.

MFC after:	2 weeks
2014-07-25 18:41:56 +00:00
Xin LI
883d80c104 As of r268075, the responsibility of rounding up buffer to optimal size have
been transferred from zio_compress_data to its caller.  Therefore, passing
the 'minblocksize' down will be a no-op.

Eliminate the parameter to reduce diff against upstream.

MFC after:	2 weeks
2014-07-25 06:53:20 +00:00
Xin LI
3d4d6b0883 Correct typo introduced with r268855.
MFC after:	10 days
X-MFC with:	r268855
2014-07-22 08:37:01 +00:00
Xin LI
b4bb49887b Reduce lock contention on the z_teardown_lock under heavily cached
read workload by splitting the single teardown rrw lock into
RRM_NUM_LOCKS (17) of them.

Read acquisitions are randomly distributed among these locks based
on curthread pointer.  Write acquisitions are going to all the
locks, which for the usage of this type of lock should be rare.

Illumos issue:
    5008 lock contention (rrw_exit) while running a read only load

MFC after:	2 weeks
2014-07-19 00:26:03 +00:00
Xin LI
82599d31fe MFV r268851:
When a sync task is waiting for a txg to complete, we should hurry it along
by increasing the number of outstanding async writes (i.e. make
vdev_queue_max_async_writes() return a larger number).

Illumos issue:
    4753 increase number of outstanding async writes when sync task is waiting

MFC after:	2 weeks
2014-07-18 22:34:01 +00:00
Xin LI
f886b6e3bc MFV r268850:
Change the interaction between the DMU and ARC so that when the DMU is
shutting down an objset, we do not evict the data from the ARC.  Instead
we simply coordinate the destruction of the DMU's data with the ARC.

The only case where we actually need to explicitly evict from the ARC is
when dbuf_rele_and_unlock() determines that the administrator has requested
that it not be kept in memory, via the primarycache/secondarycache properties.
In this case, we evict the data from the ARC by its blkptr_t, the same way
as when a block is freed we explicitly evict it from the ARC.

Illumos issue:
    4631 zvol_get_stats triggering too many reads

MFC after:	2 weeks
2014-07-18 22:04:21 +00:00
Xin LI
7882b61f60 MFV r268848:
Instead of asserting all zio's be properly aligned, only assert
on the logical ones.

Cap uberblocks at 8k, otherwise with ashift=17, there would be
only one uberblock.

This fixes a problem that zdb would trip assert on pools with
ashift >= 0xe (8k).

While there, also change the code so it only attempt to condense
space map unless the uncondensed size consumes greater than
zfs_metaslab_condense_block_threshold blocks.

Illumos issue:
  4958 zdb trips assert on pools with ashift >= 0xe

MFC after:	2 weeks
2014-07-18 20:41:40 +00:00
Xin LI
7079d5877c MFV r268714:
Improve extreme rewind import.

When doing an "extreme rewind" import ("zpool import -XF"), we attempt
to verify all data in the pool, essentially scrubbing the entire pool.
The problem is that spa_load_verify_cb() issues an unbounded number of
concurrent scrub i/os.  This can lead to all of memory being used for
these zio's, wedging the system. Like normal scrub, we need to put a
cap on the number of outstanding i/os, and have the traverse thread
block when we reach this cap.

For this purpose the cap can be very large (10,000) to optimize the
elevator algorithm.  Three kernel tunables have been added:

	vfs.zfs.spa_load_verify_maxinflight
	vfs.zfs.spa_load_verify_metadata
	vfs.zfs.spa_load_verify_data

The latter two tunables controls whether metadata and/or user data
when doing extreme rewind.

Make 'zpool import -T' imply scrub.

Make zpool import -T <txg> accept hexadecimal values for the txg when
prefixed with 0x.

Skip txg's for which there is no uberblock when doing extreme rewind.

Skip reading all user data twice by skipping prefetches when doing
extreme rewinds as we do not access via the ARC.

Illumos issues:
  4970 need controls on i/o issued by zpool import -XF
  4971 zpool import -T should accept hex values
  4972 zpool import -T implies extreme rewind, and thus a scrub
  4973 spa_load_retry retries the same txg
  4974 spa_load_verify() reads all data twice

MFC after:	2 weeks
2014-07-15 22:44:04 +00:00
Xin LI
eb75155228 MFV r268702:
Add missing *_destroy() calls in various places with ZFS.

Illumos issue:
  4975 missing mutex_destroy() calls in zfs

MFC after:	2 weeks
2014-07-15 20:32:23 +00:00
Xin LI
1b174fa1eb MFV r268455:
Use reserved space for ZFS administrative commands.

We reserve 1/2^spa_slop_shift = 1/32 or 3.125% of pool space (or 32MB at
least) for system use.  Most ZPL operations, e.g. write(2), creat(2), will
fail with ENOSPC if we fall below this.

Certain operations, e.g. file removal and most administrative actions,
still permitted until half of the slop space is used.  This would allow
users to use these operations to free up space in the pool when pool is
close to full but half of slop space is still free.

A very restricted set of operations that frees up space or change quota
are always permitted, regardless of the amount of free space.

MFC after:	 2 weeks
2014-07-09 23:14:59 +00:00
Xin LI
fdc0ee2cf5 MFV r268452:
Explicitly mark file removal transactions as "presumed to result
in a net free of space" so they will not fail with ENOSPC.

Illumos issue:	4950 files sometimes can't be removed from a full
		filesystem
MFC after:	2 weeks
2014-07-09 18:32:40 +00:00
Alexander Motin
e327a057a7 Remove IO_SYNC flag when writing extended file attributes on ZFS.
While it is possible to create and write file, modify its permissions, etc.
without ever doing sync, it looks odd that it is required for setting
extended file attributes on ZFS.  UFS does not do sync there too.

Samba uses those extended attributes to store some its data, and doing it
synchronously by many times reduces file creation performance for systems
without SLOG device.

Reviewed by:	delphij, jpaetzel, silence on fs@
MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
2014-07-08 17:26:08 +00:00
Marcel Moolenaar
e7d939bda2 Remove ia64.
This includes:
o   All directories named *ia64*
o   All files named *ia64*
o   All ia64-specific code guarded by __ia64__
o   All ia64-specific makefile logic
o   Mention of ia64 in comments and documentation

This excludes:
o   Everything under contrib/
o   Everything under crypto/
o   sys/xen/interface
o   sys/sys/elf_common.h

Discussed at: BSDcan
2014-07-07 00:27:09 +00:00
Pedro F. Giffuni
5f40879138 Merge from OpenSolaris (24-Jul-2010):
6679140 asymmetric alloc/dealloc activity can induce dynamic variable drops
6679193 dtrace_dynvar walker produces flood of dtrace_dynhash_sink

This finishes a set of merges from the older OpenSolaris releases.
Still the FreeBSD port has many differences that are difficult to
account for but that seems normal given that the kernels are different.

MFC after:	1 week
2014-07-05 15:36:17 +00:00
Pedro F. Giffuni
99d8c6efbd Merge from OpenSolaris (30-Jun-2009):
6851093 system drops to kmdb with anonymous dtrace probes + kmdb

This has no effect on FreeBSD (code is ifdef'ed) but is useful as
reference for future merges.

MFC after:	1 week
2014-07-03 19:25:24 +00:00
Pedro F. Giffuni
87e109c3e0 Merge from OpenSolaris (22-Apr-2008):
6823388 DTrace ioctl handlers must validate all structure members

MFC after:	1 week
2014-07-03 19:07:37 +00:00
Pedro F. Giffuni
e099b3a948 Merge from OpenSolaris (20-Apr-2008):
6822482 DOF validation needs to handle loadable sections flagged as unloadable

MFC after:	1 week
2014-07-03 17:36:59 +00:00
Alexander Motin
5a178afd41 Fix bug in sync control in new "dev" mode of ZVOL (r265678).
Don't check ZVOL_WCE flag, used in Solaris to control device "write cache".
It is not applicable on FreeBSD and by default set to "disable".

MFC after:	3 days
2014-07-02 21:25:32 +00:00
Pedro F. Giffuni
0b8f286e83 Merge from OpenSolaris (15-Sep-2008):
6735480 race between probe enabling and provider registration

MFC after:	1 week
2014-07-01 23:37:24 +00:00