Commit Graph

2223 Commits

Author SHA1 Message Date
Alexander Motin
9c498bd5c3 Call delist_dev() before destroy_dev_sched_cb().
destroy_dev_sched_cb() is excessively asynchronous, and during media change
retaste new provider may appear sooner then device of the previous one get
destroyed.

MFC after:	1 week
Sponsored by:	iXsystems, Inc.
2019-04-24 19:56:02 +00:00
Conrad Meyer
83efd2885e gnop(8): Nopify configuration as a kernel dump device
As a dummy / no-op dump device, to facilitate dumpon(8) testing.

Reviewed by:	markj (earlier version)
Differential Revision:	https://reviews.freebsd.org/D19991
2019-04-22 03:25:49 +00:00
Pawel Jakub Dawidek
2f07cdf871 Implement automatic online expansion of GELI providers - if the underlying
provider grows, GELI will expand automatically and will move the metadata
to the new location of the last sector.

This functionality is turned on by default. It can be turned off with the
-R flag, but it is not recommended - if the underlying provider grows and
automatic expansion is turned off, it won't be possible to attach this
provider again, as the metadata is no longer located in the last sector.

If the automatic expansion is turned off and the underlying provider grows,
GELI will only log a message with the previous size of the provider, so
recovery can be easier.

Obtained from:	Fudo Security
2019-04-03 23:57:37 +00:00
Pawel Jakub Dawidek
e6b0d5eb9f Introduce new event SIZECHANGE within GEOM system to inform about GEOM
providers mediasize changes.

While here, use GEOM nomenclature to describe providers instead of calling
them device nodes.

Obtained from:	Fudo Security
Tested in:	AWS
2019-03-30 07:24:34 +00:00
Ian Lepore
91a3f3588a Support device-independent labels for geom_flashmap slices.
While geom_flashmap has always supported label names for its slices, it does
so by appending "s.labelname" to the provider device name, meaning you still
have to know the name and unit of the hardware device to use the labels.

These changes add support for device-independent geom_flashmap labels, using
the standard geom_label infrastructure. geom_flashmap now creates a softc
struct attached to its geom, and as it creates slices it stores the label
into an array in the softc. The new geom_label_flashmap uses those labels
when tasting a geom_flashmap provider.

Differential Revision:	https://reviews.freebsd.org/D19535
2019-03-24 19:11:45 +00:00
Conrad Meyer
54533f66c9 stack(9): Drop unused API mode and comment that referenced it
Reviewed by:	markj
Differential Revision:	https://reviews.freebsd.org/D19601
2019-03-15 22:39:55 +00:00
Marcel Moolenaar
96937e3b23 Revert revision 254095
In revision 254095, gpt_entries is not set to match the on-disk
hdr_entries, but rather is computed based on available space.
There are 2 problems with this:

1.  The GPT backend respects hdr_entries and only reads and writes
    that number of partition entries.  On top of that, CRC32 is
    computed over the table that has hdr_entries elements.  When
    the common code works on what is possibly a larger number, the
    behaviour becomes inconsistent and problematic.  In particular,
    it would be possible to add a new partition that on a reboot
    isn't there anymore.
2.  The calculation of gpt_entries is based on flawed assumptions.
    The GPT specification does not dictate that sectors are layed
    out in a particular way that the available space can be
    determined by looking at LBAs.  In practice, implementations
    do the same thing, because there's no reason to do it any
    other way.  Still, GPT allows certain freedoms that can be
    exploited in some form or shape if the need arises.

PR:		229977
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D19438
2019-03-05 04:15:34 +00:00
Konstantin Belousov
e8643b01e6 Modularize xz.
Embedded lzma decompression library becomes a module usable by other
consumers, in addition to geom_uzip.

Most important code changes are
- removal of XZ_DEC_SINGLE define, we need the code to work
  with XZ_DEC_DYNALLOC;
- xz_crc32_init() call is removed from geom_uzip, xz module handles
  initialization on its own.

xz is no longer embedded into geom_uzip, instead the depend line for
the module is provided, and corresponding kernel option is added to
each MIPS kernel config file using geom_uzip.

The commit also carries unrelated cleanup by removing excess "device geom_uzip"
in places which were missed in r344479.

Reviewed by:	cem, hselasky, ray, slavash (previous versions)
Sponsored by:	Mellanox Technologies
Differential revision:	https://reviews.freebsd.org/D19266
MFC after:	3 weeks
2019-02-26 19:55:03 +00:00
Mark Johnston
3843d88ca8 Add a missing return statement to g_concat_kernel_dump().
The error occurs when upper layers attempt an out-of-bounds write.

Submitted by:	Noah Bergbauer <noah.bergbauer@tum.de>
MFC after:	1 week
2019-02-26 18:30:51 +00:00
Mark Johnston
cd2e908669 Define a constant for the maximum number of GEOM_CTL arguments.
Reviewed by:	eugen
MFC with:	r344305
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D19271
2019-02-20 17:07:08 +00:00
Mark Johnston
d4fbe32c65 Limit the number of entries allocated for a REPORT_ZONES command.
The DIOCGETZONE ioctl can be used to fetch the zone list of an SMR
drive, and the caller specifies the number of entries it wants to fetch.
Clamp the caller's request to a sane limit so that a user cannot attempt
large allocations. Callers already need to invoke the ioctl multiple
times to fetch the full list in general, so there's no harm in limiting
the number of entries returned.

Fix style while here.

admbug:		807
Reported by:	Ilja Van Sprundel <ivansprundel@ioactive.com>
Reviewed by:	asomers, ken
Tested by:	ken
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D19249
2019-02-19 21:33:02 +00:00
Mark Johnston
60a92c781d Impose a limit on the number of GEOM_CTL arguments.
Otherwise a privileged user can trigger a memory allocation of
unbounded size, or an integer overflow in the subsequent
geom_alloc_copyin() call, leading to out-of-bounds accesses.

Hard-code a large limit to circumvent this problem.

admbug:		854
Reported by:	Anonymous of the Shellphish Grill Team
Reviewed by:	ae
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D19251
2019-02-19 21:22:22 +00:00
Andriy Voskoboinyk
81df432ecf geom_uzip(4): set 'gp != NULL' assertion on top of the function
There was yet another access to this variable in g_trace() few
lines upper.

PR:		203499
Reported by:	cem
MFC after:	5 days
MFC with:	343473
2019-01-26 17:17:25 +00:00
Andriy Voskoboinyk
34fd9d7000 geom_uzip(4): move NULL pointer KASSERT check before it is dereferenced
PR:		203499
Submitted by:	<chadf@triularity.org>
MFC after:	5 days
2019-01-26 14:54:06 +00:00
Conrad Meyer
797f009d59 gmirror: Relocate DEVICE_FLAGS to adjacent lines
gmirror's sc_flags is shared between some on-disk state and some runtime
only state.  There's no real reason for that and they could probably be
split up.  Until they are, locate all of the flags for the same field
nearby each other in the source, for clarity.

No functional change.

Sponsored by:	Dell EMC Isilon
2019-01-23 16:44:21 +00:00
Mark Johnston
438622af06 Use g_handleattr() to reply to GEOM::candelete queries.
g_handleattr() fills out bp->bio_completed; otherwise, g_getattr()
returns an error in response to the query.  This caused BIO_DELETE
support to not be propagated through stacked configurations, e.g.,
a gconcat of gmirror volumes would not handle BIO_DELETE even when
the gmirrors do.  g_io_getattr() was not affected by the problem.

PR:		232676
Reported and tested by:	noah.bergbauer@tum.de
MFC after:	1 week
2019-01-02 15:52:16 +00:00
Alexander Motin
02a9923034 Switch from mutexes to atomics in GEOM_DEV I/O path.
Mutexes in I/O path there were used twice per I/O to atomically access
several variables to close and/or destroy the device on last request
completion.  I found the way to fit all required info into one integer,
suitable for atomic operations.  It opened race window on device close,
but addition of timeout to the msleep() there should cover it.

Profiling shows removal of significant spinning time on those mutexes
and IOPS increase from ~600K to >800K to NVMe on 72-core systems.

MFC after:	1 month
Sponsored by:	iXsystems, Inc.
2018-12-27 19:15:24 +00:00
Conrad Meyer
d2d82bfc90 gmirror: Remove a last-minute INVARIANTS breakage in r341840
I mistakenly added a lock assertion to this routine at the last minute
without confirming it was held during g_mirror_create.  It isn't (it isn't
even initialized yet).  Mea culpa.  Access is exclusive in both callers,
just not always by that particular lock.

Reported by:	lwhsu
X-MFC-With:	r341840, r341674
2018-12-12 18:13:56 +00:00
Conrad Meyer
23c25bd8b1 gmirror: Fix a bug introduced in r341674
r341674 inadvertently introduced a bug where newer mirror components being
tasted would clear the high sc_flags that are not controlled by component
metadata, such as G_MIRROR_DEVICE_FLAG_TASTING.  This could plausibly expose
a small window of time during STARTING where device destruction might race
with mirror component addition, probably resulting in a crash.

Reviewed by:	markj
X-MFC-With:	r341674
Differential Revision:	https://reviews.freebsd.org/D18521
2018-12-12 05:48:27 +00:00
Conrad Meyer
af7dcae0e2 gmirror: Evaluate mirror components against newest metadata copy
Re-apply r341665 with format strings fixed.

If we happen to taste a stale mirror component first, don't reject valid,
newer components that have differing metadata from the stale component
(during STARTING).  Instead, update our view of the most recent metadata as
we taste components.

Like mediasize beforehand, remove some checks from g_mirror_check_metadata
which would evict valid components due to metadata that can change over a
mirror's lifetime.  g_mirror_check_metadata is invoked long before we check
genid/syncid and decide which component(s) are newest and whether or not we
have quorum.

Before checking if we can enter RUNNING (i.e., we have quorum) after a NEW
component is added, first remove any known stale or inconsistent disks from
the mirrorset, rather than removing them *after* deciding we have quorum.
Check if we have quorum after removing these components.

Additionally, add a knob, kern.geom.mirror.launch_mirror_before_timeout, to
force gmirrors to wait out the full timeout (kern.geom.mirror.timeout)
before transitioning from STARTING to RUNNING.  This is a kludge to help
ensure all eligible, boot-time available mirror components are tasted before
RUNNING a gmirror.

Add a basic test case for STARTING -> RUNNING startup behavior around stale
genids.

PR:		232671, 232835
Submitted by:	Cindy Yang <cyang AT isilon.com> (previous version)
Reviewed by:	markj (kernel portions)
Discussed with:	asomers, Cindy Yang
Tested by:	pho
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D18062
2018-12-07 02:44:04 +00:00
Conrad Meyer
c4e87bdfc1 Revert r341665 due to tinderbox breakage
I didn't notice that some format strings were non-portable.  Will fix and
re-commit later.
2018-12-07 00:47:05 +00:00
Conrad Meyer
bc1ee0be2d gmirror: Evaluate mirror components against newest metadata copy
If we happen to taste a stale mirror component first, don't reject valid,
newer components that have differing metadata from the stale component
(during STARTING).  Instead, update our view of the most recent metadata as
we taste components.

Like mediasize beforehand, remove some checks from g_mirror_check_metadata
which would evict valid components due to metadata that can change over a
mirror's lifetime.  g_mirror_check_metadata is invoked long before we check
genid/syncid and decide which component(s) are newest and whether or not we
have quorum.

Before checking if we can enter RUNNING (i.e., we have quorum) after a NEW
component is added, first remove any known stale or inconsistent disks from
the mirrorset, rather than removing them *after* deciding we have quorum.
Check if we have quorum after removing these components.

Additionally, add a knob, kern.geom.mirror.launch_mirror_before_timeout, to
force gmirrors to wait out the full timeout (kern.geom.mirror.timeout)
before transitioning from STARTING to RUNNING.  This is a kludge to help
ensure all eligible, boot-time available mirror components are tasted before
RUNNING a gmirror.

When we are instructed to forget mirror components, bump the generation id
to avoid confusion with such stale components later.

Add a basic test case for STARTING -> RUNNING startup behavior around stale
genids.

PR:		232671, 232835
Submitted by:	Cindy Yang <cyang AT isilon.com> (previous version)
Reviewed by:	markj (kernel portions)
Discussed with:	asomers, Cindy Yang
Tested by:	pho
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D18062
2018-12-06 23:55:39 +00:00
Kirk McKusick
fb14e73cb4 Normally when an attempt is made to mount a UFS/FFS filesystem whose
superblock has a check-hash error, an error message noting the
superblock check-hash failure is printed and the mount fails. The
administrator then runs fsck to repair the filesystem and when
successful, the filesystem can once again be mounted.

This approach fails if the filesystem in question is a root filesystem
from which you are trying to boot. Here, the loader fails when trying
to access the filesystem to get the kernel to boot. So it is necessary
to allow the loader to ignore the superblock check-hash error and make
a best effort to read the kernel. The filesystem may be suffiently
corrupted that the read attempt fails, but there is no harm in trying
since the loader makes no attempt to write to the filesystem.

Once the kernel is loaded and starts to run, it attempts to mount its
root filesystem. Once again, failure means that it breaks to its prompt
to ask where to get its root filesystem. Unless you have an alternate
root filesystem, you are stuck.

Since the root filesystem is initially mounted read-only, it is
safe to make an attempt to mount the root filesystem with the failed
superblock check-hash. Thus, when asked to mount a root filesystem
with a failed superblock check-hash, the kernel prints a warning
message that the root filesystem superblock check-hash needs repair,
but notes that it is ignoring the error and proceeding. It does
mark the filesystem as needing an fsck which prevents it from being
enabled for writing until fsck has been run on it. The net effect
is that the reboot fails to single user, but at least at that point
the administrator has the tools at hand to fix the problem.

Reported by:    Rick Macklem (rmacklem@)
Discussed with: Warner Losh (imp@)
Sponsored by:   Netflix
2018-12-06 00:09:39 +00:00
Maxim Sobolev
9dcafe16d4 Another attempt to fix issue with the DIOCGDELETE ioctl(2) not
handling slightly out-of-bound requests properly (r340187).
Perform range check here rather then rely on g_delete_data() to DTRT.

The g_delete_data() would always return success for requests
starting just the next byte after providers media boundary.

MFC after:	4 weeks
2018-12-04 21:48:56 +00:00
Dag-Erling Smørgrav
cdd2df880d Add a “skip_dsn” option to g_part's bootcode verb to prevent g_part_mbr
from setting the volume serial number.  This unbreaks older boot blocks
that don't support serial numbers, and allows boot0cfg to set the serial
number itself if requested by the user.

Submitted by:	lev@, yuripv@
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D17386
2018-11-27 14:58:19 +00:00
Maxim Sobolev
de66da7374 Revert r340187, it breaks EOD (end-of-device) detection logic. Turns out,
i/o into last_sector+N is handled differently for N==1 and N>1 cases to
accomodate that, so some other approach would be needed to fix DIOCGDELETE
ioctl(2).
2018-11-07 16:28:09 +00:00
Maxim Sobolev
8948179aba Don't allow BIO_READ, BIO_WRITE or BIO_DELETE requests that are
fully beyond the end of providers media. The only exception is made
for the zero length transfers which are allowed to be just on the
boundary. Previously, any requests starting on the boundary (i.e. next
byte after the last one) have been allowed to go through.

No response from:	freebsd-geom@, phk
MFC after:		1 month
2018-11-06 15:55:41 +00:00
Mark Johnston
25c9cca757 Have gconcat advertise delete support if one of its disks does.
This follows the example set by other multi-disk GEOM classes.

PR:		232676
Tested by:	noah.bergbauer@tum.de
MFC after:	1 month
2018-10-30 00:22:14 +00:00
Eugene Grosbein
6d305ab0b2 Extend stripeoffset and stripesize of GEOMs from u_int to off_t
GEOM's stripeoffset overflows at 4 gigabyte margin (2^32)
because of its u_int type. This leads to incorrect data in the output
generated by "sysctl kern.geom.confxml" command, "graid list" etc.
when GEOM array has volumes larger than 4G, for example.

This change does not affect ABI but changes KBI. No MFC planned.

Differential Revision:	https://reviews.freebsd.org/D13426
2018-10-27 16:14:42 +00:00
Xin LI
0db665bb98 Restore backward compatibility for "attach" verb.
In r332361 and r333439, two new parameters were added to geli attach
verb using gctl_get_paraml, which requires the value to be present.
This would prevent old geli(8) binary from attaching geli(4) device
as they have no knowledge about the new parameters.

Restore backward compatibility by treating the absense of these two
values as seeing the default value supplied by userland.

PR:		232595
Reviewed by:	oshogbo
MFC after:	3 days
Differential Revision:	https://reviews.freebsd.org/D17680
2018-10-27 03:37:14 +00:00
Glen Barber
01d4e2149e MFH r338661 through r339200.
Sponsored by:	The FreeBSD Foundation
2018-10-05 17:53:47 +00:00
Alexander Motin
cf4a52cf67 Fix use-after-free in RAID0 error reporting of GEOM_RAID.
PR:		231510
Submitted by:	yangx92@hotmail.com
Approved by:	re (gjb)
MFC after:	1 week
2018-09-24 16:58:55 +00:00
Jung-uk Kim
9c40dcbe5f Make geli(8) buildable. 2018-09-19 07:08:04 +00:00
Conrad Meyer
1b0909d51a OpenCrypto: Convert sessions to opaque handles instead of integers
Track session objects in the framework, and pass handles between the
framework (OCF), consumers, and drivers.  Avoid redundancy and complexity in
individual drivers by allocating session memory in the framework and
providing it to drivers in ::newsession().

Session handles are no longer integers with information encoded in various
high bits.  Use of the CRYPTO_SESID2FOO() macros should be replaced with the
appropriate crypto_ses2foo() function on the opaque session handle.

Convert OCF drivers (in particular, cryptosoft, as well as myriad others) to
the opaque handle interface.  Discard existing session tracking as much as
possible (quick pass).  There may be additional code ripe for deletion.

Convert OCF consumers (ipsec, geom_eli, krb5, cryptodev) to handle-style
interface.  The conversion is largely mechnical.

The change is documented in crypto.9.

Inspired by
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .

No objection from:	ae (ipsec portion)
Reported by:	jhb
2018-07-18 00:56:25 +00:00
Conrad Meyer
1df7f41560 OCF: Convert consumers to the session id typedef
These were missed in the earlier r336269.

No functional change.

Sponsored by:	Dell EMC Isilon
2018-07-16 19:01:05 +00:00
Mariusz Zaborski
78f79a9a08 Let geli deal with lost devices without crashing.
PR:		162036
Submitted by:	Fabian Keil <fk@fabiankeil.de>
Obtained from:	ElectroBSD
Discussed with: pjd@
2018-07-15 18:03:19 +00:00
Warner Losh
4bae19e9b8 g_eli_key_cmp is used only in the kernel, so only define it in the
kernel.
2018-07-13 18:21:38 +00:00
Mikolaj Golub
874774c5d4 geom_gate: enable resize
Reviewed By:	pjd
Approved By:	pjd
Differential Revision:	https://reviews.freebsd.org/D11531
2018-07-13 07:08:06 +00:00
Ed Maste
76db6c8773 gpart: add EFI alias for MBR partition scheme
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D15870
2018-06-17 20:10:48 +00:00
Ed Maste
a0a8412b2a Sort geom/part mbr/ebr/ldm alias table entries
Having the table entries in alpha order simplifies future additions.

Sponsored by:	The FreeBSD Foundation
2018-06-17 20:06:27 +00:00
Mariusz Zaborski
31f7586d73 Introduce the 'n' flag for the geli attach command.
If the 'n' flag is provided the provided key number will be used to
decrypt device. This can be used combined with dryrun to verify if the key
is set correctly. This can be also used to determine which key slot we want to
change on already attached device.

Reviewed by:	allanjude
Differential Revision:	https://reviews.freebsd.org/D15309
2018-05-09 20:53:38 +00:00
Mark Johnston
bd92e6b6f5 Refactor some of the MI kernel dump code in preparation for netdump.
- Add clear_dumper() to complement set_dumper().
- Drain netdump's preallocated mbuf pool when clearing the dumper.
- Don't do bounds checking for dumpers with mediasize 0.
- Add dumper callbacks for initialization for writing out headers.

Reviewed by:	sbruno
MFC after:	1 month
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D15252
2018-05-06 00:22:38 +00:00
Mark Johnston
681554d70b Remove a redundant assertion.
MFC after:	1 week
Sponsored by:	Dell EMC Isilon
2018-05-06 00:05:03 +00:00
Mark Johnston
40e805221b Avoid dropping the topology lock in gmirror's dumpconf implementation.
Doing so introduces races which can lead to a use-after-free when
grabbing a snapshot of the GEOM mesh.

To ensure that a mirror's disk list remains stable, change its locking
protocol: both the softc lock and the topology lock are now required
to modify the list, so either lock is sufficient for traversal.

Tested by:	pho
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2018-05-06 00:03:24 +00:00
Ed Maste
b525a10ac0 gpart: add fat32lba MBR partition type
FAT32 partition with LBA addressing.

Reviewed by:	marcel
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D15266
2018-05-04 00:34:27 +00:00
Kyle Evans
74d6c131cb Annotate geom modules with MODULE_VERSION
GEOM ELI may double ask the password during boot. Once at loader time, and
once at init time.

This happens due a module loading bug. By default GEOM ELI caches the
password in the kernel, but without the MODULE_VERSION annotation, the
kernel loads over the kernel module, even if the GEOM ELI was compiled into
the kernel. In this case, the newly loaded module
purges/invalidates/overwrites the GEOM ELI's password cache, which causes
the double asking.

MFC Note: There's a pc98 component to the original submission that is
omitted here due to pc98 removal in head. This part will need to be revived
upon MFC.

Reviewed by:	imp
Submitted by:	op
Obtained from:	opBSD
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D14992
2018-04-10 19:18:16 +00:00
Mariusz Zaborski
8f1c45c20a Introduce dry run option for attaching the device.
This will allow us to verify if passphrase and key is valid without
decrypting whole device.

Reviewed by:	cem@, allanjude@
Differential Revision:	https://reviews.freebsd.org/D15000
2018-04-10 13:22:48 +00:00
Kyle Evans
2967ace894 Retire the geom_aes class
It's had a good life, but it's not really configurable and not really used.

Obtained from:	opBSD (with some changes)
Differential Revision:	https://reviews.freebsd.org/D14991
2018-04-09 17:30:30 +00:00
Brooks Davis
6469bdcdb6 Move most of the contents of opt_compat.h to opt_global.h.
opt_compat.h is mentioned in nearly 180 files. In-progress network
driver compabibility improvements may add over 100 more so this is
closer to "just about everywhere" than "only some files" per the
guidance in sys/conf/options.

Keep COMPAT_LINUX32 in opt_compat.h as it is confined to a subset of
sys/compat/linux/*.c.  A fake _COMPAT_LINUX option ensure opt_compat.h
is created on all architectures.

Move COMPAT_LINUXKPI to opt_dontuse.h as it is only used to control the
set of compiled files.

Reviewed by:	kib, cem, jhb, jtl
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D14941
2018-04-06 17:35:35 +00:00
Sean Bruno
2c385d51ce Squash error from geom by sizing ident strings to DISK_IDENT_SIZE.
Display attribute in future error strings and differentiate g_handleattr()
error messages for ease of debugging in the future.

"g_handleattr: md1 bio_length 24 strlen 31 -> EFAULT"

Reported by:	swills
Reviewed by:	imp cem avg
Sponsored by:	Limelight Networks
Differential Revision:	https://reviews.freebsd.org/D14962
2018-04-05 13:56:40 +00:00