Commit Graph

1367 Commits

Author SHA1 Message Date
Emmanuel Vadot
a71ea7bea0 pkgbase: Put zfs utilities and lib in their own package
It is useful to have zfs utilities and lib in a separate package as
it allow users to create image that can support ZFS (i.e. not with
WITHOUT_ZFS in src.conf set) without bloating the default image with
all zfs tools (for example for jails).

Differential Revision:	https://reviews.freebsd.org/D36225
2022-10-26 19:46:30 +02:00
Gordon Bergling
4bd5e6221e zfsd(8): Fix a typo in an error message
- s/faild/failed/

MFC after:	1 week
2022-10-25 12:55:42 +02:00
Mark Johnston
53a31c87df libdtrace: Fix some CPU register number definitions to match the kernel
MFC after:	1 week
2022-10-12 16:06:37 -04:00
Mark Johnston
19a847e5f2 kinst: Add a rudimentary regression test case
The test instruments a number of large, frequently called kernel
functions while generating load in the background.

MFC after:	3 months
2022-10-11 18:19:55 -04:00
Christos Margiolis
2179a159ea libdtrace: Add kinst support
kinst does not instantiate its probes automatically, it only does so on
demand via an ioctl interface implemented by /dev/kinst.  This change
modifies libdtrace to perform that work when the script references the
kinst provider, similar to the way pid provider probes are implemented.

Reviewed by:	markj
MFC after:	3 months
Sponsored by:	Google, Inc. (GSoC 2022)
Differential Revision:	https://reviews.freebsd.org/D36852
2022-10-11 18:19:08 -04:00
Mark Johnston
bdd101c4d4 dtrace: Add a "regs" variable
This allows invop-based providers (i.e., fbt and kinst) to expose the
register file of the CPU at the point where the probe fired.  It does
not work for SDT providers because their probes are implemented as plain
function calls and so don't save registers.  It's not clear what
semantics "regs" should have for them anyway.

This is akin to "uregs", which nominally provides access to the
userspace registers.  In fact, DIF already had a DIF_VAR_REGS variable
defined, it was simply unimplemented.

Usage example: print the contents of %rdi upon each call to
amd64_syscall():

    fbt::amd64_syscall:entry {printf("%x", regs[R_RDI]);}

Note that the R_* constants are defined in /usr/lib/dtrace/regs_x86.d.
Currently there are no similar definitions for non-x86 platforms.

Reviewed by:	christos
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D36799
2022-10-04 13:05:54 -04:00
John Baldwin
e67b246734 libzpool: Disable -Wuse-after-free for dbuf.c.
The debug traces for reference counting in ZFS use the pointer of the
owning object as a "tag" for references to check that when an object
drops a reference it had actually held one.  In a couple of places ZFS
drops references after freeing the owning object.  In userland GCC
realizes this is a use after free.  However, since only the value of
the pointer is used, and it isn't indirected, the use is harmless.

Reviewed by:	emaste
Differential Revision:	https://reviews.freebsd.org/D36818
2022-10-03 16:10:43 -07:00
Martin Matuska
c7046f76c2 zfs: merge openzfs/zfs@c629f0bf6
Notable upstream pull request merges:
  #13725 Fix BLAKE3 tuneable and module loading on Linux and FreeBSD
  #13756 FreeBSD: Organize sysctls
  #13773 FreeBSD: add kqfilter support for zvol cdev
  #13781 Importing from cachefile can trip assertion
  #13794 Apply arc_shrink_shift to ARC above arc_c_min
  #13798 Improve too large physical ashift handling
  #13799 Revert "Avoid panic with recordsize > 128k, raw sending and
         no large_blocks"
  #13802 Add zfs.sync.snapshot_rename
  #13831 zfs_enter rework
  #13855 zfs recv hangs if max recordsize is less than received
         recordsize

Obtained from:	OpenZFS
OpenZFS commit:	c629f0bf62
2022-09-21 14:17:13 +02:00
Jens Schweikhardt
7c20397b72 In man page cross refs to sudo, indicate it comes from ports/security/sudo. 2022-08-14 11:44:43 +02:00
Mark Johnston
6237c3e74f ctfconvert: Actually use the asprintf() helper
Fixes:	1165fc9a52 ("ctfconvert: Give bitfield types names distinct from the base type")
2022-08-02 20:49:50 -04:00
Mark Johnston
1165fc9a52 ctfconvert: Give bitfield types names distinct from the base type
CTF integers have an explicit width and so can be used to represent
bitfields.  Bitfield types emitted by ctfconvert(1) share the name of
the base integer type, so a struct field with type "unsigned int : 15"
will have a type named "unsigned int".

To avoid ambiguity when looking up types by name, add a suffix to names
of bitfield types to distinguish them from the base type.  Then, if
ctfmerge happens to order bitfield types before the corresponding base
type in a CTF file, a name lookup will return the base type, which is
always going to be the desired behaviour.

PR:		265403
Reported by:	cy
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2022-08-02 20:32:17 -04:00
Mark Johnston
6a05f14381 dtrace tests: Rename some test type names to avoid a conflict
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2022-08-02 20:32:08 -04:00
Mark Johnston
e1700a36a9 dtrace tests: Override RLIMIT_CORE for a test which triggers a core dump
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2022-08-02 20:32:04 -04:00
Kristof Provost
06016a40d6 libnvpair: install libnvpair.h
This is included by libbe/be.h.

Sponsored by:	Rubicon Communications, LLC ("Netgate")
2022-07-21 18:36:01 +02:00
Martin Matuska
a0b956f5ac zfs: merge openzfs/zfs@cb01da680
Notable upstream pull request merges:
  #12438 Avoid panic with recordsize > 128k, raw sending and no large_blocks
  #13015 Fix dnode byteswapping
  #13256 Add a "zstream decompress" subcommand
  #13555 Scrub mirror children without BPs
  #13576 Several sorted scrub optimizations
  #13579 Fix and disable blocks statistics during scrub
  #13582 Several B-tree optimizations
  #13591 Avoid two 64-bit divisions per scanned block
  #13606 Avoid memory copies during mirror scrub
  #13613 Avoid memory copy when verifying raidz/draid parity

Obtained from:	OpenZFS
OpenZFS commit:	cb01da6805
2022-07-08 00:02:36 +02:00
Brooks Davis
3bf6636512 cddl/*: add a WITH(OUT)_DTRACE option
Add an option to enable/disable DTrace without disabling ZFS.  New
architectures such as CHERI may support ZFS before they support DTrace
and the old model of WITHOUT_CDDL disabling both wasn't helpful.

For compatiblity, the CDDL option remains and WITHOUT_CDDL implies
WITHOUT_DTRACE.  WITHOUT_DTRACE also implies WITHOUT_CTF.

As part of this change, largely convert cddl/*/Makefile to using the
more compact SUBDIR.${MK_<FOO>}+= form rather than using intermediate
variables.

Reviewed by:	markj
Obtained from:	CheriBSD
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D35718
2022-07-06 14:03:48 +01:00
Kornel Dulęba
9dbacce2d4 lockstat: Fix construction of comparision predicates
Passing "0x%p" to sprintf results in double "0x" being printed.
This causes a dtrace script compilation failure when "-d" flag
is specified.
Fix that by removing the extraneous "0x".

Reviewed by:	markj
Approved by:	mw(mentor)
Obtained from:	Semihalf
Sponsored by:	Alstom
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D35690
2022-07-04 16:22:22 +02:00
Brooks Davis
711d50bd9e dtrace: Remove local mips support
Remove the stub pid probe and all the build glue.

Reviewed by:	imp, jhb
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D35541
2022-07-01 08:33:17 +01:00
Jessica Clarke
b0a8333a88 libicp_rescue: Fix build for powerpc64le
This is a repeat of 13cee53cef35 ("libicp: Fix build for powerpc64le")
but for the rescue version of the library.

Fixes:	1f1e2261e3 ("zfs: merge openzfs/zfs@deb121309")
2022-07-01 04:28:15 +01:00
Jessica Clarke
9921563f43 libicp: Fix build for powerpc64le
The C sources use the assembly routines for little-endian powerpc64, and
the assembly files have ppc64le in their names, but the guard here was
for big-endian powerpc64.

Fixes:	1f1e2261e3 ("zfs: merge openzfs/zfs@deb121309")
2022-07-01 01:30:26 +01:00
Martin Matuska
1f1e2261e3 zfs: merge openzfs/zfs@deb121309
Notable upstream pull request merges:
  #12918 Introduce BLAKE3 checksums as an OpenZFS feature
  #13553 Reduce ZIO io_lock contention on sorted scrub
  #13537 Improve sorted scan memory accounting
  #13540 AVL: Remove obsolete branching optimizations
  #13563 FreeBSD: Improve crypto_dispatch() handling

Obtained from:	OpenZFS
OpenZFS commit:	deb1213098
2022-06-23 17:49:33 +02:00
Martin Matuska
e3aa18ad71 zfs: merge openzfs/zfs@b9d98453f
Notable upstream pull request merges:
  #12321 Fix inflated quiesce time caused by lwb_tx during zil_commit()
  #13244 zstd early abort
  #13360 Verify BPs as part of spa_load_verify_cb()
  #13452 More speculative prefetcher improvements
  #13466 Expose zpool guids through kstats
  #13476 Refactor Log Size Limit
  #13484 FreeBSD: libspl: Add locking around statfs globals
  #13498 Cancel in-progress rebuilds when we finish removal
  #13499 zed: Take no action on scrub/resilver checksum errors
  #13513 Remove wrong assertion in log spacemap

Obtained from:	OpenZFS
OpenZFS commit:	b9d98453f9
2022-06-03 18:17:53 +02:00
Martin Matuska
716fd348e0 zfs: merge openzfs/zfs@c0cf6ed67
Notable upstream pull request merges:
  #10662 zvol_wait: Ignore locked zvols
  #12789 Improve log spacemap load time
  #12812 Improved zpool status output, list all affected datasets
  #13277 FreeBSD: Use NDFREE_PNBUF if available
  #13302 Make zfs_max_recordsize default to 16M
  #13311 Fix error handling in FreeBSD's get/putpages VOPs
  #13345 FreeBSD: Fix translation from ABD to physical pages
  #13373 zfs: holds: dequadratify
  #13375 Corrected edge case in uncompressed ARC->L2ARC handling
  #13388 Improve mg_aliquot math
  #13405 Reduce dbuf_find() lock contention
  #13406 FreeBSD: use zero_region instead of allocating a dedicated page

Obtained from:	OpenZFS
OpenZFS commit:	c0cf6ed679
2022-05-19 00:55:59 +02:00
Mark Johnston
5727eceabc ctf: Link CTF toolchain man pages to ctf.5
Also expand the CTF acronym to provide a bit of context.

PR:		259790
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2022-04-21 11:20:28 -04:00
Mark Johnston
45dd2eaac3 ctfdump: Remove definitions of warn() and vwarn()
The presence of the latter causes a link error when building a
statically linked ctfdump(1) because libc defines the same symbol.
libc's warn() is defined as a weak symbol and so does not cause the same
problem, but let's just use libc's version.

Reported by:	stephane rochoy <stephane.rochoy@stormshield.eu>
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2022-04-21 11:18:53 -04:00
Mark Johnston
6e563a1b60 libctf: Fix recursive descent into anonymous SOU fields
PR:		262412
Tested by:	dhw, gallatin
Fixes:		a6fb869173 ("libctf: Handle CTFv3 containers")
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2022-04-12 17:29:53 -04:00
Li-Wen Hsu
0088bc444e
DTrace test: skip flakey common.misc.t_dtrace_contrib.tst_dynopt_d in CI
PR:		237641
Sponsored by:	The FreeBSD Foundation
2022-03-24 11:37:50 +08:00
Li-Wen Hsu
e62aee66b0
DTrace test: Add a new keyword SKIPCI to gentest.sh
This is for marking a test case is flakey and should not be executed in
the CI environment.

Reviewed by:	markj
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D34635
2022-03-24 11:30:27 +08:00
Li-Wen Hsu
16e02ae401
dtrace tests: Fix expected outout for tst.system.d
This is follow up of d500a85e64

PR:		262415
Sponsored by:	The FreeBSD Foundation
2022-03-09 11:39:12 +08:00
Martin Matuska
c03c5b1c80 zfs: merge openzfs/zfs@a86e08941 (master) into main
Notable upstream pull request merges:
  #9078:  log xattr=sa create/remove/update to ZIL
  #11919: Cross-platform xattr user namespace compatibility
  #13014: Report dnodes with faulty bonuslen
  #13016: FreeBSD: Fix zvol_cdev_open locking
  #13019: spl: Don't check FreeBSD rwlocks for double initialization
  #13027: Fix clearing set-uid and set-gid bits on a file when
          replying a write
  #13031: Add enumerated vdev names to 'zpool iostat -v' and
          'zpool list -v'
  #13074: Enable encrypted raw sending to pools with greater ashift
  #13076: Receive checks should allow unencrypted child datasets
  #13098: Avoid dirtying the final TXGs when exporting a pool
  #13172: Fix ENOSPC when unlinking multiple files from full pool

Obtained from:	OpenZFS
OpenZFS commit:	a86e089415
2022-03-08 18:53:02 +01:00
Mark Johnston
3fe1f21fb3 ctf: Avoid passing a caddr_t to roundup2()
For some reason I can't reproduce this locally, but Jenkins complains.

Reported by:	Jenkins
Fixes:		bdf290cd3e ("ctf: Add v3 support to CTF tools, ctf{convert,dump,merge}")
2022-03-07 11:20:57 -05:00
Mark Johnston
cb6f722562 ctf: Fix a -Wunused-but-set-variable warning
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2022-03-07 10:43:19 -05:00
Mark Johnston
bdf290cd3e ctf: Add v3 support to CTF tools, ctf{convert,dump,merge}
ctfdump handles v2 and v3.  ctfconvert now emits only CTFv3, whereas
ctfmerge can merge v2 and v3 containers into v3 containers.

MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D34364
2022-03-07 10:43:19 -05:00
Mark Johnston
a6fb869173 libctf: Handle CTFv3 containers
In general, the patch adds indirection to minimize the amount of code
that needs to know about differences between v2 and v3.  Specifically,
some new ctf_get_ctt_* functions are added, and new LCTF_* macros are
added to use the underlying container's version to do the right thing.

CTF containers can have parent/child relationships, wherein a type ID in
one container refers to a type in the parent.  It is permitted for the
parent and child to have different versions.

MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D34363
2022-03-07 10:43:19 -05:00
Mark Johnston
994297b01b ctf: Add definitions for CTFv3
These are based on definitions added to binutils' libctf.  Specifically:
- Type IDs are now encoded in 32 bits rather than 16, changing the
  layout of ctf_type_t, ctf_array_t, ctf_member_t and ctf_lmember_t.
- Type info is encoded in 32 bits rather than 16.  The type "kind" is
  extended from 5 bits to 6, and the type "vlen" is extended from 10
  bits to 25.

The main upside is that we remove the current limit, imposed by CTFv2,
of 2^{15} distinct types in the main kernel executable.  Other limits,
such as that on the number of elements in an enum, imposed by the vlen
limit, are also raised.

This change adds v2 and v3 flavours of macros and type definitions which
differ between the two versions.  Compatibility is preserved for now by
having generic names refer to the v2 definitions, so, e.g., ctf_type_t
is still a v2 type.

No functional change intended.

Reviewed by:	Domagoj Stolfa
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D34360
2022-03-07 10:43:18 -05:00
Mark Johnston
2d5d2a986c ctf: Import ctf.h from OpenBSD
Use it instead of the existing ctf.h from OpenSolaris.  This makes it
easier to use CTF in the core kernel, and to extend the CTF format to
support wider type IDs.

The imported ctf.h is modified to depend only on _types.h, and also to
provide macros which use the "parent" bit of a type ID to refer to types
in a parent CTF container.

No functional change intended.

Reviewed by:	Domagoj Stolfa, emaste
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D34358
2022-03-07 10:43:18 -05:00
Mark Johnston
45c23c2608 libctf: Use const ctf_file_t references in string lookup routines
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2022-02-23 18:43:51 -05:00
Mark Johnston
1ef441f699 libdtrace: Add a missing newline to an error message
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2022-02-23 11:41:23 -05:00
Mark Johnston
565518046c libctf: Remove checks for CTFv1
Per commit 7db423d692 ("libctf: Rip out CTFv1 support") this support
is obsolete.

MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2022-02-23 11:41:22 -05:00
Mark Johnston
7be9a3b453 ctfconvert: Rip out STABS support
It is unused on FreeBSD and complicates some efforts to modify the CTF
format to permit wider type IDs, so remove it.  No functional change
intended.

MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2022-02-10 15:41:10 -05:00
Mark Johnston
7db423d692 libctf: Rip out CTFv1 support
CTFv1 was obsolete before libctf was imported into FreeBSD, and
ctfconvert/ctfmerge can emit only CTFv2.  Make ctf.h a bit easier to
maintain by ripping v1 support out.  No functional change intended.

MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2022-02-10 15:39:59 -05:00
Mark Johnston
2e4311906d libctf: Use ctf_type_t instead of struct ctf_type
For consistency with other CTF toolchain code.  No functional change
intended.

Fixes:	105fd928b0 ("libctf: Improve check for duplicate SOU definitions in ctf_add_type()")
MFC after:	1 week
2022-02-10 13:21:37 -05:00
Chuck Silvers
f339a3ef63 dtrace: remove unnecessary fflush()
This call was added back in the early days of dtrace porting and
no one knows why anymore.  The extra flushing causes lots of
unnecessary CPU overhead when a script produces lots of output,
as well as easily losing output because the command can't keep up.

Sponsored by:	Netflix
Reviewed by:	imp, markj
Differential Revision:	https://reviews.freebsd.org/D34216
2022-02-09 17:09:26 -08:00
Dimitry Andric
5f2aca8394 Disable clang 14 warning about bitwise operators in zstd
Parts of zstd, used in openzfs and other places, trigger a new clang 14
-Werror warning:

```
sys/contrib/zstd/lib/decompress/huf_decompress.c:889:25: error: use of bitwise '&' with boolean operands [-Werror,-Wbitwise-instead-of-logical]
                        (BIT_reloadDStreamFast(&bitD1) == BIT_DStream_unfinished)
                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

While the warning is benign, it should ideally be fixed upstream and
then vendor-imported, but for now silence it selectively.

MFC after:	3 days
2022-02-08 21:46:08 +01:00
Martin Matuska
e92ffd9b62 zfs: merge openzfs/zfs@17b2ae0b2 (master) into main
Notable upstream pull request merges:
  #12766 Fix error propagation from lzc_send_redacted
  #12805 Updated the lz4 decompressor
  #12851 FreeBSD: Provide correct file generation number
  #12857 Verify dRAID empty sectors
  #12874 FreeBSD: Update argument types for VOP_READDIR
  #12896 Reduce number of arc_prune threads
  #12934 FreeBSD: Fix zvol_*_open() locking
  #12947 lz4: Cherrypick fix for CVE-2021-3520
  #12961 FreeBSD: Fix leaked strings in libspl mnttab
  #12964 Fix handling of errors from dmu_write_uio_dbuf() on FreeBSD
  #12981 Introduce a flag to skip comparing the local mac when raw sending
  #12985 Avoid memory allocations in the ARC eviction thread

Obtained from:	OpenZFS
OpenZFS commit:	17b2ae0b24
2022-01-22 23:05:15 +01:00
Andriy Gapon
256c8c5df2 dt_unring_buf: set dtbd_oldest to the start of the first record
It was set to the start of the buffer and that can be different from the
start of teh first record because of a misalignment.

This change follows the example of dt_realloc_buf().

Reviewed by:	tsoome, markj
MFC after:	4 weeks
Differential Revision:	https://reviews.freebsd.org/D33649
2022-01-11 15:44:04 +02:00
Mark Johnston
963f5dc7a3 ctfconvert: Handle arrays of empty structs
Members with such a type will legitimately have a size of zero, so don't
emit a warning.

PR:		260818
Reviewed by:	bz
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D33700
2021-12-31 12:55:01 -05:00
Domagoj Stolfa
30ec3138ed dtrace: Disable getf() as it is broken on FreeBSD
getf() on FreeBSD calls _sx_slock(), _sx_sunlock() and fget_locked().
Furthermore, it does not set the per-core fault flag, meaning it
usually ends up in a double fault panic once getf() does get called,
especially from fbt.

Reviewing the DTrace Toolkit + a number of other scripts scattered
around FreeBSD, I have not been able to find one use of getf(). Given
how broken the implementation currently is, we disable it until it
can be implemented properly.

Also comment out a test in aggs/tst.subr.d for getf().

Reviewed by:	markj
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D33378
2021-12-17 13:10:22 -05:00
Greg V
9e9c651cac cddl: fix missing ZFS library dependencies
In 9fae47666 zfsd got a libspl dependency to avoid undefined references.
However that workaround did not help external consumers of libzfs_core.

Fix all missing dependencies lld 13 and the rtld complain about.

Reviewed by:	freqlabs, markj
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D32521
2021-11-15 09:42:07 -05:00
Martin Matuska
6ba2210ee0 zfs: merge openzfs/zfs@ec64fdb93 (master) into main
Notable upstream pull request merges:
  #12392 Avoid panic in case of pool errors and missing L2ARC
  #12448 skip snapshot in zfs_iter_mounted()
  #12516 Fix NFS and large reads on older kernels
  #12533 Fail invalid incremental recursive send gracefully
  #12569 FreeBSD: Really zero the zero page
  #12575 Reject zfs send -RI with nonexistent fromsnap
  #12602 Correct refcount_add in dmu_zfetch
  #12650 zpool should call zfs_nicestrtonum() with non-NULL handle

Obtained from:	OpenZFS
OpenZFS commit:	ec64fdb93d
2021-10-21 15:06:06 +02:00