illumos/illumos-gate@555d674d5d555d674d5dhttps://www.illumos.org/issues/10592
This is a collection of recent fixes from ZoL:
8eef997679b Error path in metaslab_load_impl() forgets to drop ms_sync_lock
928e8ad47d3 Introduce auxiliary metaslab histograms
425d3237ee8 Get rid of space_map_update() for ms_synced_length
6c926f426a2 Simplify log vdev removal code
21e7cf5da89 zdb -L should skip leak detection altogether
df72b8bebe0 Rename range_tree_verify to range_tree_verify_not_present
75058f33034 Remove unused vdev_t fields
Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Author: Serapheim Dimitropoulos <serapheim@delphix.com>
MFC after: 4 weeks
illumos/illumos-gate@663207adb1663207adb1
10601 Pool allocation classes
https://www.illumos.org/issues/10601
illumos port of ZoL Pool allocation classes. Includes at least these two
commits:
441709695 Pool allocation classes misplacing small file blocks
cc99f275a Pool allocation classes
10757 Add -gLp to zpool subcommands for alt vdev names
https://www.illumos.org/issues/10757
Port from ZoL of
d2f3e292d Add -gLp to zpool subcommands for alt vdev names
Note that a subsequent ZoL commit changed -p to -P
a77f29f93 Change full path subcommand flag from -p to -P
Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Portions contributed by: Håkan Johansson <f96hajo@chalmers.se>
Portions contributed by: Richard Yao <ryao@gentoo.org>
Portions contributed by: Chunwei Chen <david.chen@nutanix.com>
Portions contributed by: loli10K <ezomori.nozomu@gmail.com>
Author: Don Brady <don.brady@delphix.com>
11541 allocation_classes feature must be enabled to add log device
illumos/illumos-gate@c1064fd7cec1064fd7cehttps://www.illumos.org/issues/11541
After the allocation_classes feature was integrated, one can no longer add a
log device to a pool unless that feature is enabled. There is an explicit check
for this, but it is unnecessary in the case of log devices, so we should handle
this better instead of forcing the feature to be enabled.
Author: Jerry Jelinek <jerry.jelinek@joyent.com>
FreeBSD notes.
I faithfully added the new -g, -L, -P flags, but only -g does something:
vdev GUIDs are displayed instead of device names. -L, resolve symlinks,
and -P, display full disk paths, do nothing at the moment.
The use of special vdevs is backward compatible for read-only access, so
root pools should be bootable, but exercise caution.
MFC after: 4 weeks
Somehow I managed to commit the manual page with unresolved conflicts in
it.
While here, I also replaced .sp with .Pp.
MFC after: 3 weeks
X-MFC with: r354804
The DOF file output by dtrace -A contains only the loadable sections.
However, as it was created by a call to dtrace_dof_create() without
flags, the original DOF was created with the loadable sections. The
result is that the DOF includes the section headers for the unloadable
sections (COMMENTS and UTSNAME) without these sections actually being
present. This is inconsistent.
A simple change to anon_prog() ensures that the missing sections are
present in the outputted DOF. Alternatively, the call to
dtrace_dof_create() could pass the DTRACE_D_STRIP flag stripping out the
loadable sections. As the unloadable sections contain info useful for
debugging purposes they haven't been stripped.
Submitted by: Graeme Jenkinson <graeme.jenkinson@cl.cam.ac.uk>
MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D21875
Since r354804 libzpool depends on libzfs for get_system_hostid symbol.
Except for zstreamdump, all binaries linked with libzpool were already
linked with libzfs. So, zstreamdump is the only fall-out.
It's interesting that on amd64 not only I was able to successfully build
zstreamdump, I am able to run it despite having the unresolved symbol in
libzpool.
MFC after: 4 weeks
X-MFC with: r354804
10499 Multi-modifier protection (MMP)
illumos/illumos-gate@e0f1c0afa4e0f1c0afa4https://www.illumos.org/issues/10499
Port the following ZFS commits from ZoL to illumos.
379ca9cf2 Multi-modifier protection (MMP)
bbffb59ef Fix multihost stale cache file import
0d398b256 Do not initiate MMP writes while pool is suspended
10701 Correct lock ASSERTs in vdev_label_read/write
illumos/illumos-gate@58447f688d58447f688dhttps://www.illumos.org/issues/10701
Port of ZoL commit:
0091d66f4e Correct lock ASSERTs in vdev_label_read/write
At a minimum, this fixes a blown assert during an MMP test run when running on
a DEBUG build.
11770 additional mmp fixes
illumos/illumos-gate@4348eb90124348eb9012https://www.illumos.org/issues/11770
Port a few additional MMP fixes from ZoL that came in after our
initial MMP port.
4ca457b065 ZTS: Fix mmp_interval failure
ca95f70dff zpool import progress kstat
(only minimal changes from above can be pulled in right now)
060f0226e6 MMP interval and fail_intervals in uberblock
Note from the committer (me).
I do not have any use for this feature and I have not tested it. I only
did smoke testing with multihost=off.
Please be aware.
I merged the code only to make future merges easier.
Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Portions contributed by: Tim Chase <tim@chase2k.com>
Portions contributed by: sanjeevbagewadi <sanjeev.bagewadi@gmail.com>
Portions contributed by: John L. Hammond <john.hammond@intel.com>
Portions contributed by: Giuseppe Di Natale <dinatale2@llnl.gov>
Portions contributed by: Prakash Surya <surya1@llnl.gov>
Portions contributed by: Brian Behlendorf <behlendorf1@llnl.gov>
Author: Olaf Faaland <faaland1@llnl.gov>
MFC after: 4 weeks
This change is based on
r354380 8899 zpool list property documentation doesn't match actual behaviour
There is no "used" pool property, "alloc" is actually spelled
"allocated".
MFC after: 5 days
illumos/illumos-gate@9c2acf00e29c2acf00e2https://www.illumos.org/issues/10554
During the port of MMP (illumos bug 10499) from ZoL, I found this
earlier ZoL project is a prerequisite. Here is the original
description. This addition will enable us to sync an open TXG to the
main pool on demand. The functionality is similar to 'sync(2)' but
'zpool sync' will return when data has hit the main storage instead of
potentially just the ZIL as is the case with the 'sync(2)' cmd.
Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Author: Alek Pinchuk <apinchuk@datto.com>
MFC after: 3 weeks
Relnotes: possibly
Port illumos change: https://www.illumos.org/issues/11667
Move lz4.c out of zfs tree to opensolaris/common/lz4, adjust it to be
usable from kernel/stand/userland builds, so we can use just one single
source. Add lz4.h to declare lz4_compress() and lz4_decompress().
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D22037
This patch modifies the zfs_ioc_snapshot_list_next() ioctl to enable it
to take input parameters that alter the way looping through the list of
snapshots is performed. The idea here is to restrict functions that
throw away some of the snapshots returned by the ioctl to a range of
snapshots that these functions actually use. This improves efficiency
and execution speed for some rollback and send operations.
Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Signed-off-by: Alek Pinchuk <apinchuk@datto.com>
Closes#8077zfsonlinux/zfs@4c0883fb4a
MFC after: 2 weeks
illumos/illumos-gate@7931524763
FreeBSD note: some tweaking was needed to avoid a conflict with
sys/rangelock.h.
Author: Matthew Ahrens <mahrens@delphix.com>
Obtained from: illumos
MFC after: 3 weeks
Not sure if we need anything better.
Maybe we should try to port illumos libfakekernel or provide something
similar natively.
MFC after: 4 weeks
X-MFC with: r353618
illumos/illumos-gate@a0b03b161ca0b03b161chttps://www.illumos.org/issues/10330
3 recent ZoL changes in the vdev and metaslab code which we can pull over:
PR 8324 c853f382db 8324 Change target size of metaslabs from 256GB to 16GB
PR 8290 b194fab0fb 8290 Factor metaslab_load_wait() in metaslab_load()
PR 8286 419ba59145 8286 Update vdev_is_spacemap_addressable() for new spacemap
encoding
Author: Serapheim Dimitropoulos <serapheimd@gmail.com>
Obtained from: illumos, ZoL
MFC after: 2 weeks
8423 8199 7432 Implement large_dnode pool feature
7432 Large dnode pool feature
8199 multi-threaded dmu_object_alloc()
8423 Implement large_dnode pool feature
10406 large_dnode changes broke zfs recv of legacy stream
llumos/illumos-gate@54811da5ac54811da5achttps://www.illumos.org/issues/8423https://www.illumos.org/issues/8199https://www.illumos.org/issues/7432illumos/illumos-gate@811964cd9f811964cd9fhttps://www.illumos.org/issues/10406
ZoL issues:
Improved dnode allocation #6564
Clean up large dnode code #6262
Fix dnode_hold() freeing dnode behavior #8172
Fix dnode allocation race #6414, #6439
Partial: Raw sends must be able to decrease nlevels #6821, #6864
Remove unnecessary txg syncs from receive_object() Closes#7197
This updates FreeBSD large_dnode code (that was imported from ZoL) to a
version that was committed to illumos. It has some cleanups,
improvements and fixes comparing to what we have in FreeBSD now.
I think that the most significant update is 8199 multi-threaded
dmu_object_alloc().
This commit reverts r351077 that was a revert of r351074 and r351076 and
restores those changes. Required atomic operations should be available
now on all platforms where we build ZFS.
Obtained from: illumos
MFC after: 3 weeks
The feature is implemented as an extension of the existing
ZFS_IOC_RENAME ioctl. Both the userland and the DSL interfaces support
renaming only a single bookmark at a time. As of now, there is no ZCP
interface to the new functionality. I am going to add it once the DSL
interface passes a test of time.
This change picks up support for zfs_ioc_namecheck_t::ENTITY_NAME that
was added to ZoL as part of Redacted Send/Receive feature by Paul
Dagnelie <pcd@delphix.com>. This is needed to allow a bookmark name in
zc_name.
Discussed with: mahrens
Reviewed by: bcr (man page)
Sponsored by: CyberSecure
Differential Revision: https://reviews.freebsd.org/D21795
Add a small wrapper around libzfs_core's lzc_send_space() to libzfs so
that every legacy ZFS_IOC_SEND consumer, along with their userland
counterpart estimate_ioctl(), can leverage ZFS_IOC_SEND_SPACE to
request send space estimation.
The legacy functionality in zfs_ioc_send() is left untouched for
compatibility purposes.
Obtained from: ZoL
Obtained from: zfsonlinux/zfs@cf7684bc8d
Author: loli10K <ezomori.nozomu@gmail.com>
MFC after: 2 weeks
Although there is always a single stream and the total size in the
summary is always equal to the size reported for the stream, it's nice
to follow the usual output format.
MFC after: 3 days
zfsonlinux/zfs@835db58592
We have long supported estimating a size of an incremental stream from a
snapshot. We should do the same for bookmarks as well.
Obtained from: ZoL
Author: loli10K <ezomori.nozomu@gmail.com>
MFC after: 3 days
Those commands are needed to repair a FreeBSD installation so add them
to the runtime package
Reviewed by: bapt, gjb
Differential Revision: https://reviews.freebsd.org/D21498
This fixes a hole in the situation where the resume state is left from
receiving a new dataset and, so, the state is set on the dataset itself
(as opposed to %recv child).
Additionally, distinguish incremental and resume streams in error
messages.
This was also committed to ZoL:
zfsonlinux/zfs@ebeb6f23bf
MFC after: 2 weeks
Sponsored by: CyberSecure
In compare(), all error cases set the error code to EPIPE, so when an
error is set, the correct assertion to make is that the error is EPIPE,
not EINVAL.
Reviewed-by: Richard Elling <Richard.Elling@RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Ryan Moeller <ryan@freqlabs.com>
Closes#8743zfsonlinux/zfs@9dc41a769d
Submitted by: Ryan Moeller <ryan@freqlabs.com>
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D20118
They follow the conventions set by rw and sx lock probes. There is
an additional lockstat:::lockmgr-disown probe.
Update lockstat(1) to report on contention and hold events for
lockmgr locks. Document the new probes in dtrace_lockstat.4, and
deduplicate some of the existing probe descriptions.
Reviewed by: mjg
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D21355
8423 8199 7432 Implement large_dnode pool feature
8423 Implement large_dnode pool feature
8199 multi-threaded dmu_object_alloc()
7432 Large dnode pool feature
llumos/illumos-gate@54811da5ac54811da5achttps://www.illumos.org/issues/8423https://www.illumos.org/issues/8199https://www.illumos.org/issues/7432
ZoL issues:
Improved dnode allocation #6564
Clean up large dnode code #6262
Fix dnode_hold() freeing dnode behavior #8172
Fix dnode allocation race #6414, #6439
Partial: Raw sends must be able to decrease nlevels #6821, #6864
Remove unnecessary txg syncs from receive_object() Closes#7197
This updates FreeBSD large_dnode code (that was imported from ZoL) to a version
that was committed to illumos. It has some cleanups, improvements and fixes
comparing to what we have in FreeBSD now. I think that the most significant
update is 8199 multi-threaded dmu_object_alloc().
Obtained from: illumos
MFC after: 3 weeks
illumos/illumos-gate@892586e8a1892586e8a1https://www.illumos.org/issues/6585
In any pool without the extensible dataset feature flag already enabled,
creating a dataset with dedup set to use one of the new checksums would result
in the following panic as soon as any data was added:
panic[cpu0]/thread=ffffff0006761c40: feature_get_refcount(spa, feature,
&refcount) != 48 (0x30 != 0x30), file: ../../common/fs/zfs/zfeature.c line 390
ffffff0006761830 fffffffffba8fbdd ()
ffffff0006761890 zfs:feature_do_action+11a ()
ffffff00067618c0 zfs:spa_feature_incr+1e ()
ffffff0006761920 zfs:dmu_object_zapify+b7 ()
ffffff00067619b0 zfs:dsl_dataset_activate_feature+97 ()
ffffff0006761a20 zfs:dsl_dataset_sync+ba ()
ffffff0006761ab0 zfs:dsl_pool_sync+153 ()
ffffff0006761b70 zfs:spa_sync+26e ()
ffffff0006761c20 zfs:txg_sync_thread+227 ()
ffffff0006761c30 unix:thread_start+8 ()
Inspection showed that feature->fi_feature was 7, which is the value of
SPA_FEATURE_EXTENSIBLE_DATASET in the spa_feature enum.
Testing shows that the panic can be prevented by explicitly setting extensible
dataset as a dependency for the sha512, edonr, and skein feature flags.
Alternatively, the new checksums code could possibly be changed to obviate the
need for the dependency.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Richard Laager <rlaager@wiktel.com>
Approved by: Robert Mustacchi <rm@joyent.com>
Author: ilovezfs <ilovezfs@icloud.com>
Note that FreeBSD does not support ednor yet.
MFC after: 2 weeks
Many thanks to cryx-freebsd@h3q.com for reporting the problem and
submitting a fix. I have chosen to take an equivalent but textually
different patch from ZoL just to avoid increasing divergence between
OpenZFS flavours.
ZoL commit: zfsonlinux/zfse33da554c5daf0103b093f44ab5b90ad6c064c3f
Author: loli10K <ezomori.nozomu@gmail.com>
Date: Wed Sep 7 19:34:20 2016 +0200
PR: 197821
Submitted by: cryx-freebsd@h3q.com (alternative version)
Reported by: cryx-freebsd@h3q.com
Obtained from: ZoL
MFC after: 1 week
We should support removing vdev from boot pool. Update loader zfs reader
to support com.delphix:removing.
Reviewed by: allanjude
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D18901
Incorporate a fix from zol:
ab5036df1c
commit log from upstream:
Fix race in parallel mount's thread dispatching algorithm
Strategy of parallel mount is as follows.
1) Initial thread dispatching is to select sets of mount points that
don't have dependencies on other sets, hence threads can/should run
lock-less and shouldn't race with other threads for other sets. Each
thread dispatched corresponds to top level directory which may or may
not have datasets to be mounted on sub directories.
2) Subsequent recursive thread dispatching for each thread from 1)
is to mount datasets for each set of mount points. The mount points
within each set have dependencies (i.e. child directories), so child
directories are processed only after parent directory completes.
The problem is that the initial thread dispatching in
zfs_foreach_mountpoint() can be multi-threaded when it needs to be
single-threaded, and this puts threads under race condition. This race
appeared as mount/unmount issues on ZoL for ZoL having different
timing regarding mount(2) execution due to fork(2)/exec(2) of mount(8).
`zfs unmount -a` which expects proper mount order can't unmount if the
mounts were reordered by the race condition.
There are currently two known patterns of input list `handles` in
`zfs_foreach_mountpoint(..,handles,..)` which cause the race condition.
1) #8833 case where input is `/a /a /a/b` after sorting.
The problem is that libzfs_path_contains() can't correctly handle an
input list with two same top level directories.
There is a race between two POSIX threads A and B,
* ThreadA for "/a" for test1 and "/a/b"
* ThreadB for "/a" for test0/a
and in case of #8833, ThreadA won the race. Two threads were created
because "/a" wasn't considered as `"/a" contains "/a"`.
2) #8450 case where input is `/ /var/data /var/data/test` after sorting.
The problem is that libzfs_path_contains() can't correctly handle an
input list containing "/".
There is a race between two POSIX threads A and B,
* ThreadA for "/" and "/var/data/test"
* ThreadB for "/var/data"
and in case of #8450, ThreadA won the race. Two threads were created
because "/var/data" wasn't considered as `"/" contains "/var/data"`.
In other words, if there is (at least one) "/" in the input list,
the initial thread dispatching must be single-threaded since every
directory is a child of "/", meaning they all directly or indirectly
depend on "/".
In both cases, the first non_descendant_idx() call fails to correctly
determine "path1-contains-path2", and as a result the initial thread
dispatching creates another thread when it needs to be single-threaded.
Fix a conditional in libzfs_path_contains() to consider above two.
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
Signed-off-by: Tomohiro Kusumi <kusumi.tomohiro@gmail.com>
PR: 237517, 237397, 239243
Submitted by: Matthew D. Fuller <fullermd@over-yonder.net> (by email)
MFC after: 3 days
The comment property was listed in the man page twice, once under the list
of read-only properties, and again (correctly), under the list of user
editable properties.
PR: 238355
Reported by: Michael Zuo <muh.muhten@gmail.com>
Sponsored by: Klara Systems
