Commit Graph

1285 Commits

Author SHA1 Message Date
Matt Macy
b363d3d52a OpenZFS: Fix issues caused by reversed commits + rebase 2021-01-07 17:42:10 -08:00
Alex Richardson
1f6612b444 Install dtrace.h and dependencies
This makes the minimum amount of changes to allow inclusion of dtrace.h
without all the solaris compatibility headers. Installing dtrace.h allows
compiling consumers of libdtrace (e.g. https://github.com/tmetsch/python-dtrace)
without requiring a copy of the source tree.
For python-dtrace I worked around this in 58019c9a12
but being able to build the library without installed sources would be
extremely useful.

Reviewed By:	gnn
Differential Revision: https://reviews.freebsd.org/D27884
2021-01-07 09:26:21 +00:00
Andriy Gapon
b946eede04 dtrace: honor LC_NUMERIC for %'d and alike, and LC_TIME for %T
Note that the public documentation on dtrace.org fails to mention %T and
incorrectly documents %Y.  The latter actually uses format "%Y %b %e %T"
where %b is always in C locale.

Discussed with:	markj
MFC after:	1 month
Sponsored by:	Panzura
2020-12-03 11:59:40 +00:00
Jonathan T. Looney
96fbe51956 When copying types from one CTF container to another, ensure that we
encode 0-length (i.e. "") structure and union member names as offset 0.
This ensures that we don't confuse other parts of the CTF code which
expect this encoding.

This resolves a Dtrace error resolving members of anonymous structs/unions
within the (struct mbuf) type which some users were seeing after r366908.

While here, update the code in ctf_add_generic() to encode 0-length type
names as offset 0.

Reviewed by:	markj
MFC after:	2 weeks
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D27246
2020-11-20 17:26:02 +00:00
Jonathan T. Looney
3cbb4cc200 When copying types from one CTF container to another, ensure that we
always copy intrinsic data types before copying bitfields which are
based on those types. This ensures the type ordering in the destination
CTF container matches the assumption made elsewhere in the CTF code
that instrinsic data types will always appear before bitfields based on
those types.

This resolves the following error message some users have seen after
r366908:
    "/usr/lib/dtrace/ipfw.d", line 121: failed to copy type of 'ip6p':
    Conflicting type is already defined

Reviewed by:	markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D27213
2020-11-17 14:07:27 +00:00
Brooks Davis
73734d6eb1 Add missing includes of src.opts.mk
Without this "SUBDIR.${MK_TESTS}=tests" would always expand to
"SUBDIR.=tests" resulting in the tests not being built.

Sponsored by:	DARPA
2020-11-16 17:20:35 +00:00
Kyle Evans
7cc42f6d25 Do a sweep and remove most WARNS=6 settings
Repeating the default WARNS here makes it slightly more difficult to
experiment with default WARNS changes, e.g. if we did something absolutely
bananas and introduced a WARNS=7 and wanted to try lifting the default to
that.

Drop most of them; there is one in the blake2 kernel module, but I suspect
it should be dropped -- the default WARNS in the rest of the build doesn't
currently apply to kernel modules, and I haven't put too much thought into
whether it makes sense to make it so.
2020-10-01 01:10:51 +00:00
Toomas Soome
e307eb94ae loader: zfs should support bootonce an nextboot
bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.

By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.

bootonce dataset name is recorded in boot pool labels, bootenv area.

in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.

bootonce and nextboot features are usable in both UEFI and BIOS boot.

To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).

At this time, only lua loader is updated.

Sponsored by:	Netflix, Klara Inc.
Differential Revision:	https://reviews.freebsd.org/D25512
2020-09-21 09:01:10 +00:00
Mark Johnston
c6989859ae Address compiler warnings in C code used by the DTrace test suite.
Reported by:	Jenkins
MFC after:	1 week
2020-09-19 16:15:22 +00:00
Alex Richardson
79e02149fc Fix dtrace tools bootstrap on non-FreeBSD after OpenZFS import
This required surprisingly few build system changes and only two changes to the
openZFS compat headers which have been upstreamed as
https://github.com/openzfs/zfs/pull/10863

Reviewed By:	#zfs, freqlabs
Differential Revision: https://reviews.freebsd.org/D26193
2020-09-19 12:08:16 +00:00
Matt Macy
2c48331d28 MFV 2.0-rc2
- Fixes divide by zero for unusual hz
- remove cryptodev dependency
2020-09-18 23:21:24 +00:00
Andrew Turner
2a6803de1c Use MACHINE_CPUARCH when checking for arm64
Use MACHINE_CPUARCH with arm64 (aarch64) when we build code that could run
on any 64-bit Arm instruction set. This will simplify checks in downstream
consumers targeting prototype instruction sets.

The only place we check for MACHINE_ARCH == aarch64 is when building the
device tree blobs. As these are targeting current generation ISAs.

Sponsored by:	Innovate UK
Differential Revision:	https://reviews.freebsd.org/D26370
2020-09-14 16:12:28 +00:00
Alex Richardson
e17ebfd3b7 Remove -I flag for include path that doesn't exist
Found this while trying to get macOS bootstrap to work again after OpenZFS merge.

Reviewed By:	#zfs, freqlabs
Differential Revision: https://reviews.freebsd.org/D26192
2020-09-10 15:37:07 +00:00
Ryan Moeller
3fcf9dc4b7 Use MLINKS where appropriate for zfs/zpool subcommand pages
Reported by:	Julian H. Stacey
Sponsored by:	iXsystems, Inc.
2020-09-02 15:20:10 +00:00
Ryan Moeller
0667273959 Ensure zstreamdump subdir can be visited
Reported by:	Thomas Laus
Sponsored by:	iXsystems, Inc.
2020-09-01 19:02:07 +00:00
Mark Johnston
8d4b1e3b76 Correct the D definition for EINTEGRITY.
MFC after:	1 week
Sponsored by:	NetApp, Inc.
Sponsored by:	Klara, Inc.
2020-09-01 15:15:22 +00:00
Ryan Moeller
40d0fd2875 libzfs: Add missing crypto dependency
libzfs_crypto.c uses PKCS5_PBKDF2_HMAC_SHA1 from libcrypto.

Reported by:	John Kennedy
Sponsored by:	iXsystems, Inc.
2020-08-27 13:05:41 +00:00
Ryan Moeller
507cf10ad5 Move zstd sources from libzfs to libzpool
zstd is kernel code that was not supposed to be in libzfs.

libzpool provides userland shims for kernel code and is where the
zstd code needs to be included.

Reported by:	John Kennedy
Discussed with:	mmacy
Sponsored by:	iXsystems, Inc.
2020-08-26 19:03:15 +00:00
Ryan Moeller
67c97ec2d6 Tidy up libzpool Makefile
Sponsored by:	iXsystems, Inc.
2020-08-26 19:00:17 +00:00
Ryan Moeller
ec45a4d383 Install zfs-events.5
Sponsored by:	iXsystems, Inc.
2020-08-26 15:43:44 +00:00
Brandon Bergren
89509c956c [PowerPC] Apply the ppc32 GOT overflow fix to powerpcspe
powerpcspe is also a 32 bit ppc platform, and also needs to be -fPIC to
avoid overflowing the GOT.

Sponsored by:	Tag1 Consulting, Inc.
2020-08-25 18:54:10 +00:00
Matt Macy
b3fc6d6612 Fix libzfs build failures for some
In some environments adding zstd/include to the include path
would cause zstd's stdlib.h to hide the system one.
2020-08-25 03:43:52 +00:00
Matt Macy
9e5787d228 Merge OpenZFS support in to HEAD.
The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
  project quotas, encrypted datasets,
  allocation classes, vectorized raidz,
  vectorized checksums, various command line
  improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D25872
2020-08-25 02:21:27 +00:00
Mark Johnston
c612709bce Fix a typo in r364438 affecting 32-bit platforms.
Reported by:	antoine
MFC with:	r364438
2020-08-22 14:24:17 +00:00
Mark Johnston
e627909d04 Fix a typo in r364438.
Reported by:	Jenkins
MFC with:	r364438
2020-08-20 20:11:58 +00:00
Mark Johnston
d2d16e5663 Enable creation of static userspace probes in incremental builds.
To define USDT probes, dtrace -G makes use of relocations for undefined
symbols: the target address is overwritten with NOPs and the location is
recorded in the DOF section of the output object file.  To avoid link
errors, the original relocation is destroyed.  However, this means that
the same input object file cannot be processed multiple times, as
happens during incremental rebuilds.  Instead, only set the relocation
type to NONE, so that all information required to reconstruct USDT
probes is preserved.

Reported by:	bdrewery
MFC after:	3 weeks
Sponsored by:	The FreeBSD Foundation
2020-08-20 19:28:19 +00:00
Mark Johnston
0e15d9fbeb Remove non-FreeBSD ifdefs from dt_link.c.
This file is too complicated as it is and has diverged a fair bit from
illumos due to toolchain differences, so just drop unused code
(including SPARC support).

MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2020-08-20 19:27:49 +00:00
Alan Somers
02fa3d2963 zfs: fix EIO accessing dataset after resuming interrupted receive
ZFS unmounts a dataset while receiving into it and remounts it afterwards.
But if ZFS is resuming an incomplete receive, it screws up and ends up with
a dataset that is mounted, but returns EIO for every access. This commit
fixes that condition.

While the vulnerable code also exists in OpenZFS, the problem is not
reproducible there. Apparently OpenZFS doesn't unmount the destination
dataset during receive, like FreeBSD does.

PR:		248606
Reviewed by:	mmacy
MFC after:	2 weeks
Sponsored by:	Axcient
Differential Revision:	https://reviews.freebsd.org/D26034
2020-08-20 01:31:21 +00:00
Alex Richardson
fded98749a Fix libdtrace build with zsh as /bin/sh
When zsh runs in POSIX sh mode it does not support the -e flag to echo.
Use printf instead of echo to avoid the "-e" characters being printed.

Obtained from:	CheriBSD
Reviewed By:	markj
Differential Revision: https://reviews.freebsd.org/D26026
2020-08-11 16:46:54 +00:00
Alex Richardson
f57b27e3f8 Fix -DBUILD_WITH_STRICT_TMPPATH dtrace builds
Some of the scripts used for libdtrace invoke nawk instead of awk
(for example cddl/contrib/opensolaris/lib/libdtrace/common/mknames.sh).
When bootstrapping all tools, we get the nawk -> awk link while building
usr.bin/awk, but when linking/copying the dependencies from the host we
were only adding awk but not nawk.

This was silently generating invalid files when building libdtrace with
BUILD_WITH_STRICT_TMPPATH=1 since those scripts invoke nawk instead of
awk. In addition to adding the missing link this commit also adds
set -e to those scripts to catch errors like this in the future.

Reviewed By:	markj, emaste
Differential Revision: https://reviews.freebsd.org/D26025
2020-08-11 16:46:48 +00:00
Alex Richardson
ec4deee4e4 Fix cddl tools bootstrapping on macOS and Linux
Reviewed By:	brooks
Differential Revision: https://reviews.freebsd.org/D25979
2020-08-07 16:03:55 +00:00
Alex Richardson
5ac01ce026 ctfmerge: Fix missing pthread_cond_init()
This does not appear to matter on FreeBSD or Linux, but when building an
amd64 kernel on macOS I was seeing infinite loops in ctfmerge.
It turns out the loop in wip_save_work() was looping forever due to
pthread_cond_wait() always returning -EINVAL.

Reviewed By:	markj, brooks
Differential Revision: https://reviews.freebsd.org/D25973
2020-08-06 20:44:40 +00:00
Toomas Soome
722c2b4aca MFOpenZFS: Add support for boot environment data to be stored in the label
We are building new bootonce mechanism (previously zfs bootnext) and it is
based on this OpenZFS change. Since this patch is nicely self contained,
I am commiting it as is, and we can stack our changes.

Original patch description follows:

Modern bootloaders leverage data stored in the root filesystem to
enable some of their powerful features. GRUB specifically has a grubenv
file which can store large amounts of configuration data that can be
read and written at boot time and during normal operation. This allows
sysadmins to configure useful features like automated failover after
failed boot attempts. Unfortunately, due to the Copy-on-Write nature
of ZFS, the standard behavior of these tools cannot handle writing to
ZFS files safely at boot time. We need an alternative way to store
data that allows the bootloader to make changes to the data.

This work is very similar to work that was done on Illumos to enable
similar functionality in the FreeBSD bootloader. This patch is different
in that the data being stored is a raw grubenv file; this file can store
arbitrary variables and values, and the scripting provided by grub is
powerful enough that special structures are not required to implement
advanced behavior.

We repurpose the second padding area in each label to store the grubenv
file, protected by an embedded checksum. We add two ioctls to get and
set this data, and libzfs_core and libzfs functions to access them more
easily. There are no direct command line interfaces to these functions;
these will be added directly to the bootloader utilities.

Reviewed-by: Pavel Zakharov <pavel.zakharov@delphix.com>
Reviewed-by: Matthew Ahrens <mahrens@delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Closes #10009

Obtained from:	OpenZFS
Sponsored by:	Netflix, Klara Inc.
2020-08-05 14:32:20 +00:00
Yoshihiro Takahashi
8f11c99715 - Cleanups related to sparc64 removal.
- Remove remains of sparc64 files.

Reviewed by:	imp
Differential Revision:	https://reviews.freebsd.org/D25831
2020-07-28 10:58:37 +00:00
Mark Johnston
4cbba6ae24 MFOpenZFS: Fix zpool history unbounded memory usage
In original implementation, zpool history will read the whole history
before printing anything, causing memory usage goes unbounded. We fix
this by breaking it into read-print iterations.

Reviewed-by: Tom Caputi <tcaputi@datto.com>
Reviewed-by: Matt Ahrens <matt@delphix.com>
Reviewed-by: Igor Kozhukhov <igor@dilos.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Chunwei Chen <david.chen@nutanix.com>
Closes #9516

Note, this change changes the libzfs.so ABI by modifying the prototype
of zpool_get_history().  Since libzfs is effectively private to the base
system it is anticipated that this will not be a problem.

PR:		247557
Obtained from:	OpenZFS
Reported and tested by:	Sam Vaughan <samjvaughan@gmail.com>
Discussed with:	freqlabs
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D25745
openzfs/zfs@7125a109dc
2020-07-23 14:21:45 +00:00
Gordon Bergling
cb65eb43ad zpool(8): Fix a few typos regarding 'inverval' -> 'interval'
PR:		248068
Submitted by:	PauAmma <pauamma at gundo dot com>
Reviewed by:	bcr (mentor)
Approved by:	bcr (mentor)
Differential Revision:	https://reviews.freebsd.org/D25719
2020-07-20 13:01:19 +00:00
Allan Jude
af84ca2603 zpool-features(7): Note that the boot loader has support for large_blocks
Since r304321 (-current: Aug 18, 2016) and r328866 (stable/11: Feb 5, 2018)
the FreeBSD loader has supported reading from datasets with the
large_blocks feature active.

PR:		247992
Reported by:	Anton Saietskii <vsasjason@gmail.com>
MFC after:	2 weeks
Sponsored by:	Klara Inc.
2020-07-15 14:38:15 +00:00
Brooks Davis
2938ecc85c Fix a Wvoid-pointer-to-enum-cast warning missed in r359978.
This pattern is used in callbacks with void * data arguments and seems
both relatively uncommon and relatively harmless.  Silence the warning
by casting through uintptr_t.

This warning is on by default in Clang 11.

MFC after:	3 days
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D24425
2020-07-06 22:39:42 +00:00
Toomas Soome
a14844e0d6 MFOpenZFS: Add basic zfs ioc input nvpair validation
We want newer versions of libzfs_core to run against an existing
zfs kernel module (i.e. a deferred reboot or module reload after
an update).

Programmatically document, via a zfs_ioc_key_t, the valid arguments
for the ioc commands that rely on nvpair input arguments (i.e. non
legacy commands from libzfs_core). Automatically verify the expected
pairs before dispatching a command.

This initial phase focuses on the non-legacy ioctls. A follow-on
change can address the legacy ioctl input from the zfs_cmd_t.

The zfs_ioc_key_t for zfs_keys_channel_program looks like:

static const zfs_ioc_key_t zfs_keys_channel_program[] = {
       {"program",     DATA_TYPE_STRING,               0},
       {"arg",         DATA_TYPE_UNKNOWN,              0},
       {"sync",        DATA_TYPE_BOOLEAN_VALUE,        ZK_OPTIONAL},
       {"instrlimit",  DATA_TYPE_UINT64,               ZK_OPTIONAL},
       {"memlimit",    DATA_TYPE_UINT64,               ZK_OPTIONAL},
};

Introduce four input errors to identify specific input failures
(in addition to generic argument value errors like EINVAL, ERANGE,
EBADF, and E2BIG).

ZFS_ERR_IOC_CMD_UNAVAIL the ioctl number is not supported by kernel
ZFS_ERR_IOC_ARG_UNAVAIL an input argument is not supported by kernel
ZFS_ERR_IOC_ARG_REQUIRED a required input argument is missing
ZFS_ERR_IOC_ARG_BADTYPE an input argument has an invalid type

Reviewed by:	allanjude
Obtained from:	OpenZFS
Sponsored by:	Netflix, Klara Inc.
Differential Revision:	https://reviews.freebsd.org/D25393
2020-06-23 06:42:39 +00:00
Allan Jude
9598fc63e6 ZFS: Allow setting checksum=skein on boot pools
PR:		245889
Reported by:	delphij
Sponsored by:	Klara Inc.
2020-06-19 17:59:55 +00:00
Gordon Bergling
932a690c51 Add HISTORY sections to ZFS and dtrace manpage
Reviewed by:	bcr (mentor)
Approved by:	bcr (mentor)
MFC after:		7 days
Differential Revision:	https://reviews.freebsd.org/D23833
2020-06-14 05:50:28 +00:00
Konstantin Belousov
d0ca9a7fe4 Restore the binary compatibility for link_map l_addr.
Keep link_map l_addr binary layout compatible, rename l_addr to l_base
where rtld returns map base.  Provide relocbase in newly added l_addr.

This effectively reverts the patch to the initial version of D24918.

Reported by: antoine (portmgr)
Reviewed by:	jhb, markj
Tested by:	markj
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
Differential revision:	https://reviews.freebsd.org/D24946
2020-05-21 22:24:23 +00:00
Andriy Gapon
54904c38f8 zfs allow/unallow should work with numeric uid/gid
And that should work even (especially) if there is no matching user or
group name.  This change allows to see and modify delegations for
deleted groups and users.

The change is originally by Xin Li.
illumos report: https://www.illumos.org/issues/6037
OpenZFS (ZoL) PR: https://github.com/openzfs/zfs/pull/10280

Obtained from:	delphij
MFC after:	2 weeks
2020-05-12 09:04:57 +00:00
Bryan Drewery
b155807e48 dtrace tests: Support globbing for excludes
Downstream this makes skipping tests like common/ip/tst.*sctp*.ksh simpler.

Reviewed by:	vangyzen, cem, markj
Sponsored by:	Dell EMC
Differential Revision:	https://reviews.freebsd.org/D24608
2020-04-29 21:12:32 +00:00
Bryan Drewery
75a813087a Revert r360445
I did not intend to commit this yet as more work is needed for
non-amd64 kernels.
2020-04-28 18:53:49 +00:00
Bryan Drewery
6f63e88c01 ctfmerge: Assert that there is enough room for types.
Sponsord by:		Dell EMC
Differential Revision:	https://reviews.freebsd.org/D24537
2020-04-28 16:09:28 +00:00
Alan Somers
420a5a2445 zfs-program.8: fix orphan .Xr
Reported by:	phk
Reviewed by:	avg
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D24488
2020-04-18 20:55:43 +00:00
Alan Somers
35f139130a zfsd.8: fix orphan .Xr
Though ZFS is a kernel module, it has no man page in section 4.

Reported by:	phk
MFC after:	2 weeks
2020-04-18 19:47:38 +00:00
Mariusz Zaborski
23ee238fe5 zfs: Add option for forcible unmounting dataset while receiving snapshot.
Currently when the dataset is in use we can't receive snapshots.

zfs send test/1@asd | zfs recv -FM test/2
cannot unmount '/test/2': Device busy

This commits add option 'M' which attempts to forcibly unmount the
dataset.  Thanks to this we can enforce receiving snapshots in a
single step.

Note that this functionality is not supported on Linux because the
VFS will prevent active mounted filesystems from being unmounted,
even with the force option.  This is the intended VFS behavior.

Discussed-with: Pawel Jakub Dawidek <pjd@FreeBSD.org>
Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Allan Jude <allanjude@freebsd.org>
Differential Revision:	https://reviews.freebsd.org/D22306

openzfs/zfs@a57d3d45d6
2020-04-11 17:54:35 +00:00
Kyle Evans
cddf6d61d1 zfs: fix -fno-common issues
A similar (or identical?) fix has already landed in OpenZFS.

-fno-common will become the default in GCC10/LLVM11.

MFC after:	3 days
2020-03-28 17:00:38 +00:00