Commit Graph

215 Commits

Author SHA1 Message Date
Emmanuel Vadot
b5fc9e350c pkgbase: Move libicp in utilities
libicp is used by zdb zhack zinject zstream ztest libzpool.so.2 which
are all in FreeBSD-utilities.

Reviewed by:	bapt, emaste
Differential Revision:	https://reviews.freebsd.org/D29164
MFC after:	2 weeks
2021-03-16 07:12:40 +01:00
Greg V
ee21ee1572 openzfs: attach pam_zfs_key to build
This PAM module allows unlocking encrypted user home datasets when
logging in (and changing passphrase when changing the account password),
see https://github.com/openzfs/zfs/pull/9903

Also supposed to unload the key when the last session for the user is
done, but there are EBUSY issues:
https://github.com/openzfs/zfs/issues/11222#issuecomment-731897858

Submitted by:	Greg V <greg_unrelenting.technology>
Reviewed by:	mm
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D28018
2021-03-02 07:26:59 -05:00
Martin Matuska
ba27dd8be8 zfs: merge OpenZFS master-9312e0fd1
Notable upstream changes:
  778869fa1 Fix reporting of mount progress
  e7adccf7f Disable use of hardware crypto offload drivers on FreeBSD
  03e02e5b5 Fix checksum errors not being counted on repeated repair
  64e0fe14f Restore FreeBSD resource usage accounting
  11f2e9a49 Fix panic if scrubbing after removing a slog device

MFC after:	2 weeks
2021-02-22 13:01:17 +01:00
Martin Matuska
10f57cb98f zfs: bump version and install new share files
- bump version to 2.0.0-FreeBSD_gbf156c966
- install definition files for the new "-o compatibility" option
  to "zpool create"

MFC after:	2 weeks
2021-02-18 15:21:10 +01:00
Mark Johnston
8ba333e02e libdtrace: Stop relying on lex compatibility
It does not appear to be required, and as of commit 6b7e592c21
("lex: Do not let input() return 0 when end-of-file is reached") it
causes input to return 0 instead of EOF when end-of-input is reached.

PR:		253440
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2021-02-17 10:57:19 -05:00
Martin Matuska
184c1b9439 zfs: merge OpenZFS master-436ab35a5
- speed up writing to ZFS pools without ZIL devices (aa755b3)
- speed up importing ZFS pools (2d8f72d, a0e0199, cf0977a)
...

MFC after:		2 weeks
Reviewed by:		mjg (partial)
Tested by:		pho
Differential Revision:	https://reviews.freebsd.org/D28677
2021-02-16 02:46:28 +01:00
Kyle Evans
123ae3045d build: remove LIBPTHREAD/LIBTHR build options
WITHOUT_LIBTHR has been broken for a little over five years now, since the
xz 5.2.0 update introduced a hard liblzma dependency on libthr, and building
a useful system without threading support is becoming increasingly more
difficult.

Additionally, in the five plus years that it's been broken more reverse
dependencies have cropped up in libzstd, libsqlite3, and libcrypto (among
others) that make it more and more difficult to reconcile the effort needed
to fix these options.

Remove the broken options.

PR:		252760
Reviewed by:	brooks, emaste, kib
Differential Revision:	https://reviews.freebsd.org/D28263
2021-01-22 12:33:27 -06:00
Matt Macy
b363d3d52a OpenZFS: Fix issues caused by reversed commits + rebase 2021-01-07 17:42:10 -08:00
Alex Richardson
1f6612b444 Install dtrace.h and dependencies
This makes the minimum amount of changes to allow inclusion of dtrace.h
without all the solaris compatibility headers. Installing dtrace.h allows
compiling consumers of libdtrace (e.g. https://github.com/tmetsch/python-dtrace)
without requiring a copy of the source tree.
For python-dtrace I worked around this in 58019c9a12
but being able to build the library without installed sources would be
extremely useful.

Reviewed By:	gnn
Differential Revision: https://reviews.freebsd.org/D27884
2021-01-07 09:26:21 +00:00
Toomas Soome
e307eb94ae loader: zfs should support bootonce an nextboot
bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.

By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.

bootonce dataset name is recorded in boot pool labels, bootenv area.

in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.

bootonce and nextboot features are usable in both UEFI and BIOS boot.

To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).

At this time, only lua loader is updated.

Sponsored by:	Netflix, Klara Inc.
Differential Revision:	https://reviews.freebsd.org/D25512
2020-09-21 09:01:10 +00:00
Alex Richardson
79e02149fc Fix dtrace tools bootstrap on non-FreeBSD after OpenZFS import
This required surprisingly few build system changes and only two changes to the
openZFS compat headers which have been upstreamed as
https://github.com/openzfs/zfs/pull/10863

Reviewed By:	#zfs, freqlabs
Differential Revision: https://reviews.freebsd.org/D26193
2020-09-19 12:08:16 +00:00
Matt Macy
2c48331d28 MFV 2.0-rc2
- Fixes divide by zero for unusual hz
- remove cryptodev dependency
2020-09-18 23:21:24 +00:00
Andrew Turner
2a6803de1c Use MACHINE_CPUARCH when checking for arm64
Use MACHINE_CPUARCH with arm64 (aarch64) when we build code that could run
on any 64-bit Arm instruction set. This will simplify checks in downstream
consumers targeting prototype instruction sets.

The only place we check for MACHINE_ARCH == aarch64 is when building the
device tree blobs. As these are targeting current generation ISAs.

Sponsored by:	Innovate UK
Differential Revision:	https://reviews.freebsd.org/D26370
2020-09-14 16:12:28 +00:00
Alex Richardson
e17ebfd3b7 Remove -I flag for include path that doesn't exist
Found this while trying to get macOS bootstrap to work again after OpenZFS merge.

Reviewed By:	#zfs, freqlabs
Differential Revision: https://reviews.freebsd.org/D26192
2020-09-10 15:37:07 +00:00
Mark Johnston
8d4b1e3b76 Correct the D definition for EINTEGRITY.
MFC after:	1 week
Sponsored by:	NetApp, Inc.
Sponsored by:	Klara, Inc.
2020-09-01 15:15:22 +00:00
Ryan Moeller
40d0fd2875 libzfs: Add missing crypto dependency
libzfs_crypto.c uses PKCS5_PBKDF2_HMAC_SHA1 from libcrypto.

Reported by:	John Kennedy
Sponsored by:	iXsystems, Inc.
2020-08-27 13:05:41 +00:00
Ryan Moeller
507cf10ad5 Move zstd sources from libzfs to libzpool
zstd is kernel code that was not supposed to be in libzfs.

libzpool provides userland shims for kernel code and is where the
zstd code needs to be included.

Reported by:	John Kennedy
Discussed with:	mmacy
Sponsored by:	iXsystems, Inc.
2020-08-26 19:03:15 +00:00
Ryan Moeller
67c97ec2d6 Tidy up libzpool Makefile
Sponsored by:	iXsystems, Inc.
2020-08-26 19:00:17 +00:00
Brandon Bergren
89509c956c [PowerPC] Apply the ppc32 GOT overflow fix to powerpcspe
powerpcspe is also a 32 bit ppc platform, and also needs to be -fPIC to
avoid overflowing the GOT.

Sponsored by:	Tag1 Consulting, Inc.
2020-08-25 18:54:10 +00:00
Matt Macy
b3fc6d6612 Fix libzfs build failures for some
In some environments adding zstd/include to the include path
would cause zstd's stdlib.h to hide the system one.
2020-08-25 03:43:52 +00:00
Matt Macy
9e5787d228 Merge OpenZFS support in to HEAD.
The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
  project quotas, encrypted datasets,
  allocation classes, vectorized raidz,
  vectorized checksums, various command line
  improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D25872
2020-08-25 02:21:27 +00:00
Yoshihiro Takahashi
8f11c99715 - Cleanups related to sparc64 removal.
- Remove remains of sparc64 files.

Reviewed by:	imp
Differential Revision:	https://reviews.freebsd.org/D25831
2020-07-28 10:58:37 +00:00
Michael Tuexen
a357466592 sack_newdata and snd_recover hold the same value. Therefore, use only
a single instance: use snd_recover also where sack_newdata was used.

Submitted by:		Richard Scheffenegger
Differential Revision:	https://reviews.freebsd.org/D18811
2020-02-13 15:14:46 +00:00
Simon J. Gerraty
2c9a9dfc18 Update Makefile.depend files
Update a bunch of Makefile.depend files as
a result of adding Makefile.depend.options files

Reviewed by:	 bdrewery
MFC after:	1 week
Sponsored by:   Juniper Networks
Differential Revision:  https://reviews.freebsd.org/D22494
2019-12-11 17:37:53 +00:00
Toomas Soome
b0ad855d40 libzpool: use CFLAGS.lz4.c to build lz4.o and lz4.pico
Clean up this Makefile a bit.
2019-12-02 15:37:06 +00:00
Andriy Gapon
a8c08e008a MFV r354378,r354379,r354386: 10499 Multi-modifier protection (MMP)
10499 Multi-modifier protection (MMP)
illumos/illumos-gate@e0f1c0afa4
e0f1c0afa4
https://www.illumos.org/issues/10499
  Port the following ZFS commits from ZoL to illumos.
  379ca9cf2 Multi-modifier protection (MMP)
  bbffb59ef Fix multihost stale cache file import
  0d398b256 Do not initiate MMP writes while pool is suspended

10701 Correct lock ASSERTs in vdev_label_read/write
illumos/illumos-gate@58447f688d
58447f688d
https://www.illumos.org/issues/10701
  Port of ZoL commit:
  0091d66f4e Correct lock ASSERTs in vdev_label_read/write
  At a minimum, this fixes a blown assert during an MMP test run when running on
  a DEBUG build.

11770 additional mmp fixes
illumos/illumos-gate@4348eb9012
4348eb9012
https://www.illumos.org/issues/11770
  Port a few additional MMP fixes from ZoL that came in after our
  initial MMP port.
  4ca457b065 ZTS: Fix mmp_interval failure
  ca95f70dff zpool import progress kstat
  (only minimal changes from above can be pulled in right now)
  060f0226e6 MMP interval and fail_intervals in uberblock

Note from the committer (me).
I do not have any use for this feature and I have not tested it.  I only
did smoke testing with multihost=off.
Please be aware.
I merged the code only to make future merges easier.

Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Portions contributed by: Tim Chase <tim@chase2k.com>
Portions contributed by: sanjeevbagewadi <sanjeev.bagewadi@gmail.com>
Portions contributed by: John L. Hammond <john.hammond@intel.com>
Portions contributed by: Giuseppe Di Natale <dinatale2@llnl.gov>
Portions contributed by: Prakash Surya <surya1@llnl.gov>
Portions contributed by: Brian Behlendorf <behlendorf1@llnl.gov>
Author: Olaf Faaland <faaland1@llnl.gov>

MFC after:	4 weeks
2019-11-18 09:38:35 +00:00
Toomas Soome
24e1a7ac77 r354253 did miss the fact that libzpool is built as fake kernel
We build libzpool as kernel like, use _FAKE_KERNEL check to include
kernel api in libzpool.
2019-11-02 21:02:54 +00:00
Toomas Soome
e499793e76 Remove duplicate lz4 implementations
Port illumos change: https://www.illumos.org/issues/11667

Move lz4.c out of zfs tree to opensolaris/common/lz4, adjust it to be
usable from kernel/stand/userland builds, so we can use just one single
source. Add lz4.h to declare lz4_compress() and lz4_decompress().

MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D22037
2019-11-02 12:28:04 +00:00
Emmanuel Vadot
c2a7c3bedd pkgbase: Force zfs(8) and zpool(8) to be in the runtime package
Those commands are needed to repair a FreeBSD installation so add them
to the runtime package

Reviewed by:	bapt, gjb
Differential Revision:	https://reviews.freebsd.org/D21498
2019-09-05 14:07:49 +00:00
Kirk McKusick
88640c0e8b Create new EINTEGRITY error with message "Integrity check failed".
An integrity check such as a check-hash or a cross-correlation failed.
The integrity error falls between EINVAL that identifies errors in
parameters to a system call and EIO that identifies errors with the
underlying storage media. EINTEGRITY is typically raised by intermediate
kernel layers such as a filesystem or an in-kernel GEOM subsystem when
they detect inconsistencies. Uses include allowing the mount(8) command
to return a different exit value to automate the running of fsck(8)
during a system boot.

These changes make no use of the new error, they just add it. Later
commits will be made for the use of the new error number and it will
be added to additional manual pages as appropriate.

Reviewed by:    gnn, dim, brueffer, imp
Discussed with: kib, cem, emaste, ed, jilles
Differential Revision: https://reviews.freebsd.org/D18765
2019-01-17 06:35:45 +00:00
Michael Tuexen
1e88cc8b59 Add support for send, receive and state-change DTrace providers for
SCTP. They are based on what is specified in the Solaris DTrace manual
for Solaris 11.4.

Reviewed by:		0mp, dteske, markj
Relnotes:		yes
Differential Revision:	https://reviews.freebsd.org/D16839
2018-08-22 21:23:32 +00:00
Matt Macy
d12e91d584 Make dnode definition uniform on !x86
gcc4 requires -fms-extensions to accept anonymous union members
2018-08-21 03:45:09 +00:00
Kyle Evans
7920ad944b libbe(3): Move build goop back out of cddl/
Some background: in the GSoC project, libbe/Makefile lived in lib/libbe. I
created projects/bectl branch, maintained the above for all of five
minutes before I misread Makefile.inc1 and decided that it couldn't possibly
build outside of cddl/, so I kicked the Makefile out into the cddl/ build
and all was good. The misreading was of the bit where .WAIT is added to
SUBDIR after lib, libexec but prior to building bin and cddl *only during
the install targets*, which is the critical part.

Fast forward- buildworld was still broken in my branch unbeknownst to me
because I didn't nuke my OBJDIR. Combing through Makefile.inc1 eventually
revealed the necessary magic to make sure that libbe's dependencies are
specified well enough, and it becomes clear what needs done to make a
non-cddl/ build work. This is an interesting prospect, because the build
split is kind of annoying to work with.

IGNORE_PRAGMA is added to avoid dropping WARNS by one more. This was
previously pulled in via cddl/Makefile.inc.
2018-08-18 03:20:59 +00:00
Kyle Evans
f25a4e58ec libbe(3): Remove -v from LDFLAGS
-v is clearly not needed for linking, and it adds extra verbose information
that is not necessary.
2018-08-18 03:08:54 +00:00
Kyle Evans
f2fdf2a1dc libbe(3)/bectl(8): Remove now-redundant include paths
These were previously necessary because the libnvpair and libzfs_core
includes were not installed into the SYSROOT, being a part of the copies
target in include/Makefile rather than being installed with the library.

This was fixed in r337696 and the headers are now installed properly, so we
may let go of the cruft.
2018-08-13 05:01:19 +00:00
Kyle Evans
ce33c57d6c Use INCS for non-sys/ libnvpair and libzfs_core includes
While nothing was wrong with libnvpair.h, libzfs_core.h was only guarded by
MK_CDDL rather than MK_CDDL && MK_ZFS. Rather than ugl'if'ying
include/Makefile to impose the extra restriction, just move the non-sys/
includes into INCS with the respect lib builds.

This has the added bonus of allowing third party packagers to try and split
these libs out of the FreeBSD-runtime package, if they are so inclined.

The sys/ include was left alone- generally userland libraries shouldn't
install kernel headers.

MFC after:	1 week
2018-08-13 03:38:32 +00:00
Kyle Evans
35d2028fb8 libbe(3)/bectl(8): More SYSROOT/GCC build fixes
- Missing include path
- Fully specify libzfs's dependencies (except for deps pulled in by other
  deps) in Makefile.inc1
- Drop WARNS back down to 2 for libbe(3). I do this with much hesitation,
  but the libzfs headers are apparently a hot warning-filled mess as far as
  GCC 4.2 is concerned.
2018-08-11 22:45:39 +00:00
Kyle Evans
14b841d4a8 MFH @ r337607, in preparation for boarding 2018-08-11 04:26:29 +00:00
Alexander Motin
7fca1b93c4 Do not blindly include illumos kernel headers instead of user-space.
It is not needed now, and I doubt it much helped at all, creating more
confusions then good.
2018-08-02 18:55:55 +00:00
Michael Tuexen
7bda966394 Add a dtrace provider for UDP-Lite.
The dtrace provider for UDP-Lite is modeled after the UDP provider.
This fixes the bug that UDP-Lite packets were triggering the UDP
provider.
Thanks to dteske@ for providing the dwatch module.

Reviewed by:		dteske@, markj@, rrs@
Relnotes:		yes
Differential Revision:	https://reviews.freebsd.org/D16377
2018-07-31 22:56:03 +00:00
Kyle Evans
b29bf2f84e libbe(3)/be(8): Drop WARNS overrides, fix all fallout
Based on the idea that we shouldn't have all-new library and utility going
into base that need WARNS=1...

- Decent amount of constification
- Lots of parentheses
- Minor other nits
2018-07-25 15:14:35 +00:00
Kyle Evans
268af06d3e Normalize bectl(8)/libbe(3) Makefiles, remove Makefile copyright/license
Approved by:	hselaskey
2018-07-24 19:55:02 +00:00
Kyle Evans
70a11a8eea libbe(3): Add to cddl build, adjust src.libnames.mk as needed 2018-07-24 15:42:23 +00:00
Michael Tuexen
d073fd606c Add support for TCP state names used by Solaris.
For compatibility, add the TCP state names used by Solaris
and given in the Dtrace Guide available at
https://docs.oracle.com/cd/E37838_01/html/E61035/glhgu.html#OSDTGglhmv

Reviewed by:		markj@
Sponsored by:		Netflix, Inc.
Differential Revision:	https://reviews.freebsd.org/D16269
2018-07-14 17:12:04 +00:00
Michael Tuexen
7575d3df5b The IP, TCP, and UDP provider report IP addresses as strings.
In some cases, the required information is not available and the
UDP provider reported an empty string in this case and the IP
and TCP provider reported a NULL pointer.

This patch changes the value provided in this case to the string
"<unknown>". This make the behaviour consistent and in-line with
the behaviour of Solaris.

Reviewed by:		markj@, dteske@, gnn@
Differential Revision:	https://reviews.freebsd.org/D15855
2018-06-18 18:35:29 +00:00
Sean Bruno
8d7181d1e0 Unbreak dtrace runtime for udp after svn r334719 SO_REUSEPORT commit.
Submitted by:	Johannes Lundberg <johalun0@gmail.com>
Sponsored by:	Limeligght Networks
2018-06-07 15:27:07 +00:00
Sean Bruno
1a43cff92a Load balance sockets with new SO_REUSEPORT_LB option.
This patch adds a new socket option, SO_REUSEPORT_LB, which allow multiple
programs or threads to bind to the same port and incoming connections will be
load balanced using a hash function.

Most of the code was copied from a similar patch for DragonflyBSD.

However, in DragonflyBSD, load balancing is a global on/off setting and can not
be set per socket. This patch allows for simultaneous use of both the current
SO_REUSEPORT and the new SO_REUSEPORT_LB options on the same system.

Required changes to structures:
Globally change so_options from 16 to 32 bit value to allow for more options.
Add hashtable in pcbinfo to hold all SO_REUSEPORT_LB sockets.

Limitations:
As DragonflyBSD, a load balance group is limited to 256 pcbs (256 programs or
threads sharing the same socket).

This is a substantially different contribution as compared to its original
incarnation at svn r332894 and reverted at svn r332967.  Thanks to rwatson@
for the substantive feedback that is included in this commit.

Submitted by:	Johannes Lundberg <johalun0@gmail.com>
Obtained from:	DragonflyBSD
Relnotes:	Yes
Sponsored by:	Limelight Networks
Differential Revision:	https://reviews.freebsd.org/D11003
2018-06-06 15:45:57 +00:00
Sean Bruno
7875017ca9 Revert r332894 at the request of the submitter.
Submitted by:	Johannes Lundberg <johalun0_gmail.com>
Sponsored by:	Limelight Networks
2018-04-24 19:55:12 +00:00
Sean Bruno
7b7796eea5 Load balance sockets with new SO_REUSEPORT_LB option
This patch adds a new socket option, SO_REUSEPORT_LB, which allow multiple
programs or threads to bind to the same port and incoming connections will be
load balanced using a hash function.

Most of the code was copied from a similar patch for DragonflyBSD.

However, in DragonflyBSD, load balancing is a global on/off setting and can not
be set per socket. This patch allows for simultaneous use of both the current
SO_REUSEPORT and the new SO_REUSEPORT_LB options on the same system.

Required changes to structures
Globally change so_options from 16 to 32 bit value to allow for more options.
Add hashtable in pcbinfo to hold all SO_REUSEPORT_LB sockets.

Limitations
As DragonflyBSD, a load balance group is limited to 256 pcbs
(256 programs or threads sharing the same socket).

Submitted by:	Johannes Lundberg <johanlun0@gmail.com>
Sponsored by:	Limelight Networks
Differential Revision:	https://reviews.freebsd.org/D11003
2018-04-23 19:51:00 +00:00
Devin Teske
16e39c48a3 Consistent casing for fallback SIGCHLD (s/Unknown/unknown/) 2018-02-26 00:04:21 +00:00