Commit Graph

1312 Commits

Author SHA1 Message Date
Martin Matuska
48b4fe0503 zfs: attach zpool_influxdb to build
From the zpool_influxdb.8 manual page:
  zpool_influxdb produces InfluxDB-line-protocol-compatible metrics from
  zpools.  Like the zpool command, zpool_influxdb reads the current pool
  status and statistics.  Unlike the zpool command which is intended for
  humans, zpool_influxdb formats the output in the InfluxDB line protocol.
  The expected use is as a plugin to a metrics collector or aggregator,
  such as Telegraf.

zpool_influxdb is installed into /usr/libexec/zfs/

Differential revision:	https://reviews.freebsd.org/D31094
MFC after:	3 days
2021-07-07 20:15:12 +02:00
Ryan Moeller
53b438b242 zfsd: Check for error from zpool_vdev_online
Onlining a vdev can fail. Log the error if it does.

Reviewed by:	mav, asomers
MFC after:	1 week
Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D30882
2021-07-06 16:00:18 +00:00
Emmanuel Vadot
79ac3c12a7 pkgbase: Put dtrace in its own package
While dtrace is usefull some people might not want it.

Differential Revision:	https://reviews.freebsd.org/D30752
Sponsored by:	Diablotin Systems
2021-06-19 17:49:31 +02:00
Domagoj Stolfa
a877965fa3 dtrace: fix an out of bound read and a NULL pointer increment
In dt_cc.c when the provider is an empty string, accessing
strlen(pdp->dtpd_provider) - 1 will result in a pdp->dtpd_provider[-1]
access.

Similarly, in dt_ident.c, if p2 is a NULL pointer, doing a p2++ on it is
undefined behaviour.

Reviewed by:	markj
MFC after:	1 week
Sponsored by:	Google
Differential Revision:	https://reviews.freebsd.org/D30778
2021-06-17 13:52:32 -04:00
Jessica Clarke
d5c1296234 spl: Unbreak crossbuild after f20893853e
Summary:
f20893853e made getexecname no longer guarded by BOOTSTRAPPING, but it
does not build on non-FreeBSD as it tries to use KERN_PROC_PATHNAME and
related constants.

Reviewers: mm, imp

Reviewed By: imp

Subscribers: imp

Differential Revision: https://reviews.freebsd.org/D30763
2021-06-15 01:23:04 +01:00
Martin Matuska
44877c8db0 zfs: change zstream -> zstreamdump link into a symlink 2021-06-13 05:40:18 +02:00
Martin Matuska
3ff01b231d zfs: merge openzfs/zfs@afa7b3484 (master) into main
Notable upstream pull request merges:
  #12149: Multiple man-pages: Move to appropriate section
  #12158: Re-embed multilist_t storage
  #12191: Convert non-libzpool libraries to -fvisibility=hidden
  #12196: Unify manpage makefiles, move pages to better sexions, ...
  #12210: libzutil: import filtering optimisation
  #12212: Remove pool io kstats

Obtained from:	OpenZFS
OpenZFS commit:	afa7b34845
2021-06-13 04:38:47 +02:00
Warner Losh
7aa2e90176 spl: Fix gcc6 build
-Wno-error= is only a clang flag, restrict its use to only clang.

Sponsored by:		Netflix
2021-06-10 21:28:52 -06:00
Martin Matuska
47ddbfae27 zfs: unbreak build with clang 12
Change -Wno-error-atomic-alignment to -Wno-error=atomic-alignment
in the Makefile of libspl.

Reported by:	Ed Maste <emaste@FreeBSD.org>
Fix by:		Alexander Richardson <arichardson@FreeBSD.org>
2021-06-11 01:34:46 +02:00
Martin Matuska
f20893853e zfs: unbreak build broken in 160388166 2021-06-08 22:21:11 +02:00
Martin Matuska
1603881667 zfs: merge openzfs/zfs@75b4cbf62 (master) into main
Notable upstream pull request merges:
  #11710 Allow zfs to send replication streams with missing snapshots
  #11751 Avoid taking global lock to destroy zfsdev state
  #11786 Ratelimit deadman zevents as with delay zevents
  #11803 ZFS traverse_visitbp optimization to limit prefetch
  #11813 Allow pool names that look like Solaris disk names
  #11822 Atomically check and set dropped zevent count
  #11822 Don't scale zfs_zevent_len_max by CPU count
  #11833 Refactor zfsdev state init/destroy to share common code
  #11837 zfs get -p only outputs 3 columns if "clones" property is empty
  #11843 libzutil: zfs_isnumber(): return false if input empty
  #11849 Use dsl_scan_setup_check() to setup a scrub
  #11861 Improvements to the 'compatibility' property
  #11862 cmd/zfs receive: allow dry-run (-n) to check property args
  #11864 receive: don't fail inheriting (-x) properties on wrong dataset type
  #11877 Combine zio caches if possible
  #11881 FreeBSD: use vnlru_free_vfsops if available
  #11883 FreeBSD: add support for lockless symlink lookup
  #11884 FreeBSD: add missing seqc write begin/end around zfs_acl_chown_setattr
  #11896 Fix crash in zio_done error reporting
  #11905 zfs-send(8): Restore sorting of flags
  #11926 FreeBSD: damage control racing .. lookups in face of mkdir/rmdir
  #11930 vdev_mirror: don't scrub/resilver devices that can't be read
  #11938 Fix AVX512BW Fletcher code on AVX512-but-not-BW machines
  #11955 zfs get: don't lookup mount options when using "-s local"
  #11956 libzfs: add keylocation=https://, backed by fetch(3) or libcurl
  #11959 vdev_id: variable not getting expanded under map_slot()
  #11966 Scale worker threads and taskqs with number of CPUs
  #11994 Clean up use of zfs_log_create in zfs_dir
  #11997 FreeBSD: Don't force xattr mount option
  #11997 FreeBSD: Implement xattr=sa
  #11997 FreeBSD: Use SET_ERROR to trace xattr name errors
  #11998 Simplify/fix dnode_move() for dn_zfetch
  #12003 FreeBSD: Initialize/destroy zp->z_lock
  #12010 Fix dRAID self-healing short columns
  #12033 Revert "Fix raw sends on encrypted datasets when copying back snapshots"
  #12040 Reinstate the old zpool read label logic as a fallback
  #12046 Improve scrub maxinflight_bytes math
  #12049 FreeBSD: avoid memory allocation in arc_prune_async
  #12052 FreeBSD: incorporate changes to the VFS_QUOTACTL(9) KPI
  #12061 Fix dRAID sequential resilver silent damage handling
  #12072 Let zfs diff be more permissive
  #12077 FreeBSD: Retry OCF ENOMEM errors.
  #12088 Propagate vdev state due to invalid label corruption
  #12091 libzfs: On FreeBSD, use MNT_NOWAIT with getfsstat
  #12097 FreeBSD: Update dataset_kstats for zvols in dev mode
  #12104 FreeBSD boot code reminder after zpool upgrade
  #12114 Introduce write-mostly sums

Obtained from:	OpenZFS
OpenZFS commit:	75b4cbf625
2021-06-08 16:52:44 +02:00
Alan Somers
fc3ba3e9fa Delete obsolete Solaris compat files
These files have been unused ever since the OpenSolaris import

Sponsored by:	Axcient
MFC after:	2 weeks
Reviewed By:	freqlabs
Differential Revision: https://reviews.freebsd.org/D30371
2021-05-24 08:48:47 -06:00
Mark Johnston
d500a85e64 dtrace tests: Fix tst.system.d after ping/ping6 unification
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2021-04-23 10:28:09 -04:00
Domagoj Stolfa
7653f9317b dtrace: Document the libdir, nolibs and syslibdir options
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D29541
2021-04-02 17:42:38 -04:00
Mark Johnston
3c065eeaa7 libctf: Adjust logic to match upstream after 410556f1f
No functional change intended.

Suggested by:	jrtc27
MFC after:	1 week
2021-03-27 20:37:12 -04:00
Domagoj Stolfa
410556f1f1 libctf: Fix an out-of-bounds read in ctf_lookup_by_name()
When prefixes such as struct, union, etc. are compared with the current
type (e.g. struct foo), a comparison is made with the prefix.  The code
currently assumes that every type is a valid C type with a prefix,
however at times, garbage ends up in this function causing an
unpredictable crash with DTrace due to the isspace(*p) call or
subsequent calls. An example that I've seen of this is the letter 's'
being passed in, comparing true with struct as the comparison size was
(q - p) == 1, but then we increment p with the length of "struct",
resulting in an out of bounds read.

Reviewed by:	markj
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D29435
2021-03-27 14:04:12 -04:00
Emmanuel Vadot
b5fc9e350c pkgbase: Move libicp in utilities
libicp is used by zdb zhack zinject zstream ztest libzpool.so.2 which
are all in FreeBSD-utilities.

Reviewed by:	bapt, emaste
Differential Revision:	https://reviews.freebsd.org/D29164
MFC after:	2 weeks
2021-03-16 07:12:40 +01:00
Greg V
ee21ee1572 openzfs: attach pam_zfs_key to build
This PAM module allows unlocking encrypted user home datasets when
logging in (and changing passphrase when changing the account password),
see https://github.com/openzfs/zfs/pull/9903

Also supposed to unload the key when the last session for the user is
done, but there are EBUSY issues:
https://github.com/openzfs/zfs/issues/11222#issuecomment-731897858

Submitted by:	Greg V <greg_unrelenting.technology>
Reviewed by:	mm
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D28018
2021-03-02 07:26:59 -05:00
Ryan Moeller
2ae79aa362 Install links for zpool feature compat aliases
The alias links were missed when this feature was introduced to the
FreeBSD build system in 10f57cb98f.

Reviewed by:	mm
MFC after:	1 week
Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D28925
2021-02-25 14:16:09 -05:00
Martin Matuska
ba27dd8be8 zfs: merge OpenZFS master-9312e0fd1
Notable upstream changes:
  778869fa1 Fix reporting of mount progress
  e7adccf7f Disable use of hardware crypto offload drivers on FreeBSD
  03e02e5b5 Fix checksum errors not being counted on repeated repair
  64e0fe14f Restore FreeBSD resource usage accounting
  11f2e9a49 Fix panic if scrubbing after removing a slog device

MFC after:	2 weeks
2021-02-22 13:01:17 +01:00
Mark Johnston
150fc89a12 libdtrace: Trivial style fixes to force dt_lex.c to be regenerated
After commit 8ba333e02e ("libdtrace: Stop relying on lex
compatibility"), there have been several reports of incremental
buildworlds failing since make does not know that dt_lex.c needs to be
regenerated, and I want to avoid this when merging to stable/13.

MFC with:	8ba333e02e
2021-02-19 21:51:18 -05:00
Martin Matuska
10f57cb98f zfs: bump version and install new share files
- bump version to 2.0.0-FreeBSD_gbf156c966
- install definition files for the new "-o compatibility" option
  to "zpool create"

MFC after:	2 weeks
2021-02-18 15:21:10 +01:00
Mark Johnston
8ba333e02e libdtrace: Stop relying on lex compatibility
It does not appear to be required, and as of commit 6b7e592c21
("lex: Do not let input() return 0 when end-of-file is reached") it
causes input to return 0 instead of EOF when end-of-input is reached.

PR:		253440
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2021-02-17 10:57:19 -05:00
Martin Matuska
184c1b9439 zfs: merge OpenZFS master-436ab35a5
- speed up writing to ZFS pools without ZIL devices (aa755b3)
- speed up importing ZFS pools (2d8f72d, a0e0199, cf0977a)
...

MFC after:		2 weeks
Reviewed by:		mjg (partial)
Tested by:		pho
Differential Revision:	https://reviews.freebsd.org/D28677
2021-02-16 02:46:28 +01:00
Kyle Evans
123ae3045d build: remove LIBPTHREAD/LIBTHR build options
WITHOUT_LIBTHR has been broken for a little over five years now, since the
xz 5.2.0 update introduced a hard liblzma dependency on libthr, and building
a useful system without threading support is becoming increasingly more
difficult.

Additionally, in the five plus years that it's been broken more reverse
dependencies have cropped up in libzstd, libsqlite3, and libcrypto (among
others) that make it more and more difficult to reconcile the effort needed
to fix these options.

Remove the broken options.

PR:		252760
Reviewed by:	brooks, emaste, kib
Differential Revision:	https://reviews.freebsd.org/D28263
2021-01-22 12:33:27 -06:00
Ryan Libby
c14e17a49c dtrace tests: fix prototypes for gcc build
- quiet -Wstrict-prototypes
 - provide prototypes for weak aliases

Reviewed by:	markj
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D28036
2021-01-10 21:53:15 -08:00
Mark Johnston
d00431a7bd libdtrace: Format USDT symbols correctly based on symbol binding
Before we did not handle weak symbols correctly, sometimes resulting in
link errors from dtrace -G when processing object files where functions
with weak aliases contain USDT probes.

Reported by:	rlibby
Tested by:	rlibby
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2021-01-10 17:58:38 -05:00
Matt Macy
b363d3d52a OpenZFS: Fix issues caused by reversed commits + rebase 2021-01-07 17:42:10 -08:00
Alex Richardson
1f6612b444 Install dtrace.h and dependencies
This makes the minimum amount of changes to allow inclusion of dtrace.h
without all the solaris compatibility headers. Installing dtrace.h allows
compiling consumers of libdtrace (e.g. https://github.com/tmetsch/python-dtrace)
without requiring a copy of the source tree.
For python-dtrace I worked around this in 58019c9a12
but being able to build the library without installed sources would be
extremely useful.

Reviewed By:	gnn
Differential Revision: https://reviews.freebsd.org/D27884
2021-01-07 09:26:21 +00:00
Andriy Gapon
b946eede04 dtrace: honor LC_NUMERIC for %'d and alike, and LC_TIME for %T
Note that the public documentation on dtrace.org fails to mention %T and
incorrectly documents %Y.  The latter actually uses format "%Y %b %e %T"
where %b is always in C locale.

Discussed with:	markj
MFC after:	1 month
Sponsored by:	Panzura
2020-12-03 11:59:40 +00:00
Jonathan T. Looney
96fbe51956 When copying types from one CTF container to another, ensure that we
encode 0-length (i.e. "") structure and union member names as offset 0.
This ensures that we don't confuse other parts of the CTF code which
expect this encoding.

This resolves a Dtrace error resolving members of anonymous structs/unions
within the (struct mbuf) type which some users were seeing after r366908.

While here, update the code in ctf_add_generic() to encode 0-length type
names as offset 0.

Reviewed by:	markj
MFC after:	2 weeks
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D27246
2020-11-20 17:26:02 +00:00
Jonathan T. Looney
3cbb4cc200 When copying types from one CTF container to another, ensure that we
always copy intrinsic data types before copying bitfields which are
based on those types. This ensures the type ordering in the destination
CTF container matches the assumption made elsewhere in the CTF code
that instrinsic data types will always appear before bitfields based on
those types.

This resolves the following error message some users have seen after
r366908:
    "/usr/lib/dtrace/ipfw.d", line 121: failed to copy type of 'ip6p':
    Conflicting type is already defined

Reviewed by:	markj
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D27213
2020-11-17 14:07:27 +00:00
Brooks Davis
73734d6eb1 Add missing includes of src.opts.mk
Without this "SUBDIR.${MK_TESTS}=tests" would always expand to
"SUBDIR.=tests" resulting in the tests not being built.

Sponsored by:	DARPA
2020-11-16 17:20:35 +00:00
Kyle Evans
7cc42f6d25 Do a sweep and remove most WARNS=6 settings
Repeating the default WARNS here makes it slightly more difficult to
experiment with default WARNS changes, e.g. if we did something absolutely
bananas and introduced a WARNS=7 and wanted to try lifting the default to
that.

Drop most of them; there is one in the blake2 kernel module, but I suspect
it should be dropped -- the default WARNS in the rest of the build doesn't
currently apply to kernel modules, and I haven't put too much thought into
whether it makes sense to make it so.
2020-10-01 01:10:51 +00:00
Toomas Soome
e307eb94ae loader: zfs should support bootonce an nextboot
bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.

By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.

bootonce dataset name is recorded in boot pool labels, bootenv area.

in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.

bootonce and nextboot features are usable in both UEFI and BIOS boot.

To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).

At this time, only lua loader is updated.

Sponsored by:	Netflix, Klara Inc.
Differential Revision:	https://reviews.freebsd.org/D25512
2020-09-21 09:01:10 +00:00
Mark Johnston
c6989859ae Address compiler warnings in C code used by the DTrace test suite.
Reported by:	Jenkins
MFC after:	1 week
2020-09-19 16:15:22 +00:00
Alex Richardson
79e02149fc Fix dtrace tools bootstrap on non-FreeBSD after OpenZFS import
This required surprisingly few build system changes and only two changes to the
openZFS compat headers which have been upstreamed as
https://github.com/openzfs/zfs/pull/10863

Reviewed By:	#zfs, freqlabs
Differential Revision: https://reviews.freebsd.org/D26193
2020-09-19 12:08:16 +00:00
Matt Macy
2c48331d28 MFV 2.0-rc2
- Fixes divide by zero for unusual hz
- remove cryptodev dependency
2020-09-18 23:21:24 +00:00
Andrew Turner
2a6803de1c Use MACHINE_CPUARCH when checking for arm64
Use MACHINE_CPUARCH with arm64 (aarch64) when we build code that could run
on any 64-bit Arm instruction set. This will simplify checks in downstream
consumers targeting prototype instruction sets.

The only place we check for MACHINE_ARCH == aarch64 is when building the
device tree blobs. As these are targeting current generation ISAs.

Sponsored by:	Innovate UK
Differential Revision:	https://reviews.freebsd.org/D26370
2020-09-14 16:12:28 +00:00
Alex Richardson
e17ebfd3b7 Remove -I flag for include path that doesn't exist
Found this while trying to get macOS bootstrap to work again after OpenZFS merge.

Reviewed By:	#zfs, freqlabs
Differential Revision: https://reviews.freebsd.org/D26192
2020-09-10 15:37:07 +00:00
Ryan Moeller
3fcf9dc4b7 Use MLINKS where appropriate for zfs/zpool subcommand pages
Reported by:	Julian H. Stacey
Sponsored by:	iXsystems, Inc.
2020-09-02 15:20:10 +00:00
Ryan Moeller
0667273959 Ensure zstreamdump subdir can be visited
Reported by:	Thomas Laus
Sponsored by:	iXsystems, Inc.
2020-09-01 19:02:07 +00:00
Mark Johnston
8d4b1e3b76 Correct the D definition for EINTEGRITY.
MFC after:	1 week
Sponsored by:	NetApp, Inc.
Sponsored by:	Klara, Inc.
2020-09-01 15:15:22 +00:00
Ryan Moeller
40d0fd2875 libzfs: Add missing crypto dependency
libzfs_crypto.c uses PKCS5_PBKDF2_HMAC_SHA1 from libcrypto.

Reported by:	John Kennedy
Sponsored by:	iXsystems, Inc.
2020-08-27 13:05:41 +00:00
Ryan Moeller
507cf10ad5 Move zstd sources from libzfs to libzpool
zstd is kernel code that was not supposed to be in libzfs.

libzpool provides userland shims for kernel code and is where the
zstd code needs to be included.

Reported by:	John Kennedy
Discussed with:	mmacy
Sponsored by:	iXsystems, Inc.
2020-08-26 19:03:15 +00:00
Ryan Moeller
67c97ec2d6 Tidy up libzpool Makefile
Sponsored by:	iXsystems, Inc.
2020-08-26 19:00:17 +00:00
Ryan Moeller
ec45a4d383 Install zfs-events.5
Sponsored by:	iXsystems, Inc.
2020-08-26 15:43:44 +00:00
Brandon Bergren
89509c956c [PowerPC] Apply the ppc32 GOT overflow fix to powerpcspe
powerpcspe is also a 32 bit ppc platform, and also needs to be -fPIC to
avoid overflowing the GOT.

Sponsored by:	Tag1 Consulting, Inc.
2020-08-25 18:54:10 +00:00
Matt Macy
b3fc6d6612 Fix libzfs build failures for some
In some environments adding zstd/include to the include path
would cause zstd's stdlib.h to hide the system one.
2020-08-25 03:43:52 +00:00
Matt Macy
9e5787d228 Merge OpenZFS support in to HEAD.
The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
  project quotas, encrypted datasets,
  allocation classes, vectorized raidz,
  vectorized checksums, various command line
  improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D25872
2020-08-25 02:21:27 +00:00