When we're creating new providers for each of the partitions, add
aliases to the geom before we create the provider so when geom_dev
tastes the provider, the aliases are in place so the proper /dev
entries are created. So foo5p6 gets created as an alias for bar5p6
when foo is an alias for bar in the geom we're partitioning with
g_part. This also copies aliases from the container geom (eg disk) to
the label geom (the disk with GPT partitioning) so that aliases nest
properly.
Differential Revision: https://reviews.freebsd.org/D11873
Add an alias name list to geoms. Use them in geom_dev to create
aliases. Previously, geom_dev would create an device node for the name
of the geom. Now, additional nodes are created pointing back to the
primary node with make_dev_alias_p. Aliases must be in place on the
geom before any tasting occurs.
Differential Revision: https://reviews.freebsd.org/D11873
in the flush_queue:
1 2 3 4 5 6 7 8 9 10
and another 10 bio's go into the flush queue after only the first five
bio's are removed from the flush queue, the queue should look like:
6 7 8 9 10 11 12 13 14 15 16 17 18 19 20,
but because of the bug we end up with
6 11 12 13 14 15 16 17 18 19 20 7 8 9 10.
So the sequence of the bio's is damaged in the flush queue (and
therefore in the journal on disk !). This error can be triggered by
ffs_snapshot() when a block is read with readblock() and gjournal finds
this block in the broken flush queue before it goes to the correct
active queue.
The fix is to place all new blocks at the end of the queue.
Submitted by: Dr. Andreas Longwitz <longwitz@incore.de>
Discussed with: kib
MFC after: 1 week
system having over 4GB RAM. That's due to:
1) the limit being u_int instead of u_long like vm.kmem_size (the limit is
half of vm.kmem_size by default for amd64);
2) sysctl handler g_journal_cache_limit_sysctl() using u_int instead of u_long.
The fix is to replace u_int with u_long for the kern.geom.journal.cache.limit
sysctl variable.
PR: 198500
Submitted by: Dr. Andreas Longwitz <longwitz@incore.de>
Reported by: Eugene Grosbein
Discussed with: kib
MFC after: 1 week
RPI1-B, Alix and APU2 boards as well as NanoBSD with the following message:
vnode_pager_generic_getpages_done: I/O read error 5
Seems the breakage was because it was missed to include acr in glabel update.
Reported by: Peter Blok <pblok@bsd4all.org>,
madpilot, imp and trasz.
Reviewed by: trasz
Tested by: Peter Blok and madpilot.
MFC after: 3 days.
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D11365
Add -o [no]verify option to mdconfig (and document in man page.)
Implement GEOM attribute MNT::verified to ask md if the backing vnode is
verified.
Check for MNT::verified in cd9660 mount to flag the mount as MNT_VERIFIED if
the underlying device has been verified.
Reviewed by: rwatson
Approved by: sjg (mentor)
Obtained from: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D2902
During gmirror startup, if component mirrors are found to be dirty as is
typical after a system crash, the mirrors are synchronized to the mirror
with highest priority. However if a gmirror starts without all of its
mirrors present, for example because of some transient delays during
tasting, the remaining mirrors must be synchronized before they may become
active.
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Before this change it was impossible to set number of PKCS#5v2 iterations,
required to set passphrase, if it has two keys and never had any passphrase.
Due to present metadata format limitations there are still cases when number
of iterations can not be changed, but now it works in cases when it can.
PR: 218512
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D10338
At this point we have not rendezvous'ed with the mirror worker thread, and
I/O may still be in flight. Various I/O completion paths expect to be able
to obtain a reference to the mirror softc from the GEOM, so setting it to
NULL may result in various NULL pointer dereferences if the mirror is
stopped with -f or the kernel is shut down while a mirror is
synchronizing. The worker thread will clear the softc pointer before
exiting.
Tested by: pho
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
We are otherwise susceptible to a race with a concurrent teardown of the
mirror provider, causing the I/O to be left uncompleted after the mirror
started withering.
Tested by: pho
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Regular I/O requests may be blocked by concurrent synchronization requests
targeted to the same LBAs, in which case they are moved to a holding queue
until the conflicting I/O completes. We therefore want to stop
synchronization before completing pending I/O in g_mirror_destroy_provider()
since this ensures that blocked I/O requests are completed as well.
Tested by: pho
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Entries may be removed and freed if an I/O error occurs during mirror
synchronization, so we cannot assume that all entries of ds_bios are
valid.
Also ensure that a synchronization BIO's array index is preserved after
a successful write.
Reported and tested by: pho
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
This patch adds a general mechanism for providing encryption keys to the
kernel from the boot loader. This is intended to enable GELI support at
boot time, providing a better mechanism for passing keys to the kernel
than environment variables. It is designed to be extensible to other
applications, and can easily handle multiple encrypted volumes with
different keys.
This mechanism is currently used by the pending GELI EFI work.
Additionally, this mechanism can potentially be used to interface with
GRUB, opening up options for coreboot+GRUB configurations with completely
encrypted disks.
Another benefit over the existing system is that it does not require
re-deriving the user key from the password at each boot stage.
Most of this patch was written by Eric McCorkle. It was extended by
Allan Jude with a number of minor enhancements and extending the keybuf
feature into boot2.
GELI user keys are now derived once, in boot2, then passed to the loader,
which reuses the key, then passes it to the kernel, where the GELI module
destroys the keybuf after decrypting the volumes.
Submitted by: Eric McCorkle <eric@metricspace.net> (Original Version)
Reviewed by: oshogbo (earlier version), cem (earlier version)
MFC after: 3 weeks
Relnotes: yes
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D9575
In GELI, anywhere we are zeroing out possibly sensitive data, like
the metadata struct, the metadata sector (both contain the encrypted
master key), the user key, or the master key, use explicit_bzero.
Didn't touch the bzero() used to initialize structs.
Reviewed by: delphij, oshogbo
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D9809
A request may be queued while the queue lock is dropped when the mirror is
being destroyed. The corresponding wakeup would be lost, possibly resulting
in an apparent hang of the mirror worker thread.
Tested by: pho (part of a larger patch)
MFC after: 1 week
Sponsored by: Dell EMC Isilon
The worker thread will destroy the mirror provider as part of its teardown
sequence. The call made sense in the initial revision of gmirror, but
became unnecessary in r137248.
Tested by: pho (part of a larger diff)
MFC afteR: 2 weeks
Sponsored by: Dell EMC Isilon
- Don't execute any of g_mirror_shutdown_post_sync() when panicking. We
cannot safely idle the mirror or stop synchronization in that state, and
the current attempts to do so complicate debugging of gmirror itself.
- Check for a non-NULL panicstr instead of using SCHEDULER_STOPPED(). The
latter was added for use in the locking primitives.
Reviewed by: mav, pjd
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
gpart(8) has functionality to change the label of an GPT partition.
This functionality works like it should, however, after a label change
the /dev/gpt/ entries remain unchanged. glabel(8) status output remains
unchanged. The change only takes effect after a reboot.
PR: 162690
Submitted by: sub.mesa@gmail, Ben RUBSON <ben.rubson@gmail.com>, ae
Reviewed by: allanjude, bapt, bcr
MFC after: 6 weeks.
Differential Revision: https://reviews.freebsd.org/D9935
with geom_flashmap(4) and teach it about MMC for slicing enhanced
user data area partitions. The FDT slicer still is the default for
CFI, NAND and SPI flash on FDT-enabled platforms.
- In addition to a device_t, also pass the name of the GEOM provider
in question to the slicers as a single device may provide more than
provider.
- Build a geom_flashmap.ko.
- Use MODULE_VERSION() so other modules can depend on geom_flashmap(4).
- Remove redundant/superfluous GEOM routines that either do nothing
or provide/just call default GEOM (slice) functionality.
- Trim/adjust includes
Submitted by: jhibbits (RouterBoard bits)
Reviewed by: jhibbits
The PBKDF2 in sys/geom/eli/pkcs5v2.c is around half the speed it could be
GELI's PBKDF2 uses a simple benchmark to determine a number of iterations
that will takes approximately 2 seconds. The security provided is actually
half what is expected, because an attacker could use the optimized
algorithm to brute force the key in half the expected time.
With this change, all newly generated GELI keys will be approximately 2x
as strong. Previously generated keys will talk half as long to calculate,
resulting in faster mounting of encrypted volumes. Users may choose to
rekey, to generate a new key with the larger default number of iterations
using the geli(8) setkey command.
Security of existing data is not compromised, as ~1 second per brute force
attempt is still a very high threshold.
PR: 202365
Original Research: https://jbp.io/2015/08/11/pbkdf2-performance-matters/
Submitted by: Joe Pixton <jpixton@gmail.com> (Original Version), jmg (Later Version)
Reviewed by: ed, pjd, delphij
Approved by: secteam, pjd (maintainer)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D8236
Don't start switcher kproc until the first GEOM is created.
Reviewed by: pjd
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D8576
Some g_raid tasters attempt metadata reads in multiples of the provider
sectorsize. Reads larger than MAXPHYS are invalid, so detect and abort
in such situations.
Spiritually similar to r217305 / PR 147851.
PR: 214721
Sponsored by: Dell EMC Isilon
With clang 4.0.0, I'm getting the following warnings:
sys/geom/vinum/geom_vinum_state.c:186:7: error: logical not is only
applied to the left hand side of this bitwise operator
[-Werror,-Wlogical-not-parentheses]
if (!flags & GV_SETSTATE_FORCE)
^ ~
The logical not operator should obiously be called after masking.
Reviewed by: mav, pfg
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D9093
Changes include modifications in kernel crash dump routines, dumpon(8) and
savecore(8). A new tool called decryptcore(8) was added.
A new DIOCSKERNELDUMP I/O control was added to send a kernel crash dump
configuration in the diocskerneldump_arg structure to the kernel.
The old DIOCSKERNELDUMP I/O control was renamed to DIOCSKERNELDUMP_FREEBSD11 for
backward ABI compatibility.
dumpon(8) generates an one-time random symmetric key and encrypts it using
an RSA public key in capability mode. Currently only AES-256-CBC is supported
but EKCD was designed to implement support for other algorithms in the future.
The public key is chosen using the -k flag. The dumpon rc(8) script can do this
automatically during startup using the dumppubkey rc.conf(5) variable. Once the
keys are calculated dumpon sends them to the kernel via DIOCSKERNELDUMP I/O
control.
When the kernel receives the DIOCSKERNELDUMP I/O control it generates a random
IV and sets up the key schedule for the specified algorithm. Each time the
kernel tries to write a crash dump to the dump device, the IV is replaced by
a SHA-256 hash of the previous value. This is intended to make a possible
differential cryptanalysis harder since it is possible to write multiple crash
dumps without reboot by repeating the following commands:
# sysctl debug.kdb.enter=1
db> call doadump(0)
db> continue
# savecore
A kernel dump key consists of an algorithm identifier, an IV and an encrypted
symmetric key. The kernel dump key size is included in a kernel dump header.
The size is an unsigned 32-bit integer and it is aligned to a block size.
The header structure has 512 bytes to match the block size so it was required to
make a panic string 4 bytes shorter to add a new field to the header structure.
If the kernel dump key size in the header is nonzero it is assumed that the
kernel dump key is placed after the first header on the dump device and the core
dump is encrypted.
Separate functions were implemented to write the kernel dump header and the
kernel dump key as they need to be unencrypted. The dump_write function encrypts
data if the kernel was compiled with the EKCD option. Encrypted kernel textdumps
are not supported due to the way they are constructed which makes it impossible
to use the CBC mode for encryption. It should be also noted that textdumps don't
contain sensitive data by design as a user decides what information should be
dumped.
savecore(8) writes the kernel dump key to a key.# file if its size in the header
is nonzero. # is the number of the current core dump.
decryptcore(8) decrypts the core dump using a private RSA key and the kernel
dump key. This is performed by a child process in capability mode.
If the decryption was not successful the parent process removes a partially
decrypted core dump.
Description on how to encrypt crash dumps was added to the decryptcore(8),
dumpon(8), rc.conf(5) and savecore(8) manual pages.
EKCD was tested on amd64 using bhyve and i386, mipsel and sparc64 using QEMU.
The feature still has to be tested on arm and arm64 as it wasn't possible to run
FreeBSD due to the problems with QEMU emulation and lack of hardware.
Designed by: def, pjd
Reviewed by: cem, oshogbo, pjd
Partial review: delphij, emaste, jhb, kib
Approved by: pjd (mentor)
Differential Revision: https://reviews.freebsd.org/D4712
It is quite specific mode of operation without storing on-disk metadata.
It can be useful in some cases in combination with some external control
tools handling mirror creation and disks hot-plug.
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.
Upstream the BUF_TRACKING and FULL_BUF_TRACKING buffer debugging code.
This can be handy in tracking down what code touched hung bios and bufs
last. The full history is especially useful, but adds enough bloat that
it shouldn't be enabled in release builds.
Function names (or arbitrary string constants) are tracked in a
fixed-size ring in bufs. Bios gain a pointer to the upper buf for
tracking. SCSI CCBs gain a pointer to the upper bio for tracking.
Reviewed by: markj
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8366
Normally gmirror allows colliding requests to proceed whenever a
synchronization request completes and advances to the next offset. However
if an I/O request collides with one of the final g_mirror_syncreqs, nothing
releases it once synchronization completes, resulting in an apparent I/O
hang. The same problem can occur if synchronization is aborted by an
I/O error. Therefore, be sure to requeue pending requests when
mirror synchronization is stopped for any reason.
While here, remove some dead code from g_mirror_regular_release().
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Introduce internal counter to track opens. Using provider's counters is
not very successfull after calling g_wither_provider().
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.
When a syncid bump is pending, any write to the mirror results in the
updated syncid being written to each component's metadata block. However,
the update was only being performed after the writes to the mirror
componenents were queued. Instead, synchronously update the metadata block
first.
MFC after: 3 weeks
Sponsored by: Dell EMC Isilon
Consider a mirror with two components, m1 and m2. Suppose a hardware error
results in the removal of m2, with m1's genid bumped. Suppose further that
a replacement mirror component m3 is created and synchronized, after which
the system is shut down uncleanly. During a subsequent bootup, if gmirror
tastes m1 and m2 first, m2 will be removed from the mirror because it is
broken, but the mirror will be started without bumping the syncid on m1
because all elements of the mirror are accounted for. Then m3 will be
added to the already-running mirror with the same syncid as m1, so the
components will not be synchronized despite the unclean shutdown.
Handle this scenario by bumping the syncid of healthy components if any
broken mirrors are discovered during mirror startup.
MFC after: 3 weeks
Sponsored by: Dell EMC Isilon
The g_io_schedule_up() gets its "if" condition swapped to make
it more similar to g_io_schedule_down().
Suggested by: mav@
Reviewed by: mav@
MFC after: 1 month
Such errors can occur as the result of a write error or because the disk
backing the mirror element was removed. They result in a generation ID bump
on all active elements of the mirror, so we can safely disconnect the mirror
component rather than destroy it.
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D7750
These are useful for testing changes to I/O error handling, and for
reproducing existing bugs in a controlled manner. The fail points are
g_mirror_regular_request_read
g_mirror_regular_request_write
g_mirror_sync_request_read
g_mirror_sync_request_write
g_mirror_metadata_write
They all effectively allow one to inject an error value into the bio_error
field of a corresponding BIO request as it is being completed.
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
report the size only after first opening. And due to the events are
asynchronous, some consumers can receive this event too late and
this confuses them. This partially restores previous behaviour, and
at the same time this should fix the problem, when already opened
provider loses resize event.
PR: 211028
MFC after: 3 weeks
when it is being opened. This should fix the possible loss of a resize
event when disk capacity changed.
PR: 211028
Reported by: Dexuan Cui <decui at microsoft dot com>
MFC after: 3 weeks
superblock, allowing provider to be bit bigger, i.e. have some
extra padding after the FS image. That in some cases might be
a side-effect of using CLOOP format which enforces certain block
size and trying to compress image that is not exactly the number
of those blocks in size. The UFS itself does not have any issues
mounting such padded file systems, so it's what GEOM_LABEL should
do.
Submitted by: @mizhka_gmail.com
Differential Revision: https://reviews.freebsd.org/D6208
necessary because CLOOP format lacks explicit EOF or length, so that
in the presence of padding or when the CLOOP is put onto a larger
partition upper level provider size may be larger. Bound amount
of extra data that we might touch to the max length of the compressed
block and detect zero-padding in the last cluster, which when
sector is all-zero might cause us to emit bogus I/O error after
decompression of that fails. To not make code any more complicated
that it needs to be deal with it in lazy-manner, i.e. when we
first access that specific cluster.
This change also fixes stupid mistake in the LZMA code, inherited
from geom_lzma, which does not share length of the output buffer
buffer with the decompression routine, so that in the presence
of corrupted or purposedly tailored data may easily cause heap
overflow and kernel memory corruption.
Beef up validation of the CLOOP TOC by checking that lengths of
all but the last compressed clusters match upper limit set by
the decompressor and improve some error diagnostic output while
I am here.
2.Add kern.geom.uzip.attach_to tunable to artifically limit
attaching uzip to certain devices in the dev tree only.
For example the following only makes us attaching to the
GPT labels:
kern.geom.uzip.attach_to="gpt/*"
3.Add kern.geom.uzip.noattach_to, which does opposite to the (2)
above, i.e. prevents geom_uzip from tasting / attaching to
providers matching some pattern. By default we don't attach
to our own kind, i.e. kern.geom.uzip.noattach_to="*.uzip".
It saves us quite some CPU cycles, esp on low-end embedded
systems.
Approved by: re (gjb)
Differential Revision: https://reviews.freebsd.org/D7013
The GEOM disk d_mtx is only acquired on disk creation and destruction.
It is a good candidate for replacement with a pool mutex. This eliminates
the mutex initialization and teardown and the mutex and name variables
themselves from struct disk.
sys/geom/geom_disk.h:
Take d_mtx and d_mtx_name out of struct disk.
sys/geom/geom_disk.c:
Use mtx_pool_lock() and mtx_pool_unlock() to guard the disk
initialization state instead of a dedicated mutex.
This allows removing the initialization and destruction of
d_mtx.
sys/sys/param.h:
Bump __FreeBSD_version to 1100119 for the change to struct disk.
Suggested by: jhb
Sponsored by: Spectra Logic
Approved by: re (gjb)
This will result in lock recursion and is more generally incorrect since
the completion handlers will just reinsert the BIOs into the queue we're
trying to drain.
Reviewed by: imp, ngie
Approved by: re (gjb)
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D6908
after the underlying device went away.
The problem was that callers who queue the GEOM resize provider
event didn't check to make sure that the provider had not been
withered. For the other equivalent case, g_new_provider_event(),
the code checks to see whether the provider has been withered
before queueing a g_new_provider_event() to the event thread.
In some cases, a resize provider event would come through after
the provider had been withered and all of the existing consumers
had been orphaned. When the resize event triggered a taste of
the provider, that would attach a new consumer to the now
withered provider. The wither washer (g_wither_washer() would
never be able to completely tear down the GEOM because of the
consumers that were hanging around.
The solution was to check the G_PF_WITHER provider flag before
queueing the g_resize_provider_event(), and add an assert to
g_resize_provider_event() to insure that it isn't called on a
withered provider.
sys/geom/geom_subr.c:
In g_resize_provider(), don't try to continue if the
G_PF_WITHER flag is set.
In g_resize_provider_event(), add an assert that the
G_PF_WITHER flag is not set.
In g_access(), if a provider has an error, print out the
name of the provider with the error.
Sponsored by: Spectra Logic
Approved by: re (marius)
MFC after: 3 days
device is gone.
The problem was that when disk_gone() is called, if the GEOM disk
creation process has not yet happened, the withering process
couldn't start.
We didn't record any state in the GEOM disk code, and so the d_gone()
callback to the da(4) driver never happened.
The solution is to track the state of the creation process, and
initiate the withering process from g_disk_create() if the disk is
being created.
This change does add fields to struct disk, and so I have bumped
DISK_VERSION.
geom_disk.c: Track where we are in the disk creation process,
and check to see whether our underlying disk has
gone away or not.
In disk_gone(), set a new d_goneflag variable that
g_disk_create() can check to see if it needs to
clean up the disk instance.
geom_disk.h: Add a mutex to struct disk (for internal use) disk
init level, and a gone flag.
Bump DISK_VERSION because the size of struct disk has
changed and fields have been added at the beginning.
Sponsored by: Spectra Logic
Approved by: re (marius)
otherwise the system will hang.
This is a temporarily least intrusive crutch to get certain panicing systems
dumping. The proper fix should question is g_mirror_destroy() should be called
on a panicing system at all.
Discussed with: mav
ZFS's configuration needs to be updated whenever the physical path for a
device changes, but not when a new device is introduced. This is because new
devices necessarily cause config updates, but only if they are actually
accepted into the pool.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c
Split vdev_geom_set_physpath out of vdev_geom_attrchanged. When
setting the vdev's physical path, only request a config update if
the physical path has changed. Don't request it when opening a
device for the first time, because the config sync will happen
anyway upstack.
sys/geom/geom_dev.c
Split g_dev_set_physpath and g_dev_set_media out of
g_dev_attrchanged
Submitted by: will, asomers
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D6428
outlived their usefulness. This allows to remove drop/pickup Giant
wrappers around GEOM calls.
Discussed with: alfred, imp, phk
Sponsored by: The FreeBSD Foundation
This change includes support for SCSI SMR drives (which conform to the
Zoned Block Commands or ZBC spec) and ATA SMR drives (which conform to
the Zoned ATA Command Set or ZAC spec) behind SAS expanders.
This includes full management support through the GEOM BIO interface, and
through a new userland utility, zonectl(8), and through camcontrol(8).
This is now ready for filesystems to use to detect and manage zoned drives.
(There is no work in progress that I know of to use this for ZFS or UFS, if
anyone is interested, let me know and I may have some suggestions.)
Also, improve ATA command passthrough and dispatch support, both via ATA
and ATA passthrough over SCSI.
Also, add support to camcontrol(8) for the ATA Extended Power Conditions
feature set. You can now manage ATA device power states, and set various
idle time thresholds for a drive to enter lower power states.
Note that this change cannot be MFCed in full, because it depends on
changes to the struct bio API that break compatilibity. In order to
avoid breaking the stable API, only changes that don't touch or depend on
the struct bio changes can be merged. For example, the camcontrol(8)
changes don't depend on the new bio API, but zonectl(8) and the probe
changes to the da(4) and ada(4) drivers do depend on it.
Also note that the SMR changes have not yet been tested with an actual
SCSI ZBC device, or a SCSI to ATA translation layer (SAT) that supports
ZBC to ZAC translation. I have not yet gotten a suitable drive or SAT
layer, so any testing help would be appreciated. These changes have been
tested with Seagate Host Aware SATA drives attached to both SAS and SATA
controllers. Also, I do not have any SATA Host Managed devices, and I
suspect that it may take additional (hopefully minor) changes to support
them.
Thanks to Seagate for supplying the test hardware and answering questions.
sbin/camcontrol/Makefile:
Add epc.c and zone.c.
sbin/camcontrol/camcontrol.8:
Document the zone and epc subcommands.
sbin/camcontrol/camcontrol.c:
Add the zone and epc subcommands.
Add auxiliary register support to build_ata_cmd(). Make sure to
set the CAM_ATAIO_NEEDRESULT, CAM_ATAIO_DMA, and CAM_ATAIO_FPDMA
flags as appropriate for ATA commands.
Add a new get_ata_status() function to parse ATA result from SCSI
sense descriptors (for ATA passthrough over SCSI) and ATA I/O
requests.
sbin/camcontrol/camcontrol.h:
Update the build_ata_cmd() prototype
Add get_ata_status(), zone(), and epc().
sbin/camcontrol/epc.c:
Support for ATA Extended Power Conditions features. This includes
support for all features documented in the ACS-4 Revision 12
specification from t13.org (dated February 18, 2016).
The EPC feature set allows putting a drive into a power power mode
immediately, or setting timeouts so that the drive will
automatically enter progressively lower power states after various
idle times.
sbin/camcontrol/fwdownload.c:
Update the firmware download code for the new build_ata_cmd()
arguments.
sbin/camcontrol/zone.c:
Implement support for Shingled Magnetic Recording (SMR) drives
via SCSI Zoned Block Commands (ZBC) and ATA Zoned Device ATA
Command Set (ZAC).
These specs were developed in concert, and are functionally
identical. The primary differences are due to SCSI and ATA
differences. (SCSI is big endian, ATA is little endian, for
example.)
This includes support for all commands defined in the ZBC and
ZAC specs.
sys/cam/ata/ata_all.c:
Decode a number of additional ATA command names in ata_op_string().
Add a new CCB building function, ata_read_log().
Add ata_zac_mgmt_in() and ata_zac_mgmt_out() CCB building
functions. These support both DMA and NCQ encapsulation.
sys/cam/ata/ata_all.h:
Add prototypes for ata_read_log(), ata_zac_mgmt_out(), and
ata_zac_mgmt_in().
sys/cam/ata/ata_da.c:
Revamp the ada(4) driver to support zoned devices.
Add four new probe states to gather information needed for zone
support.
Add a new adasetflags() function to avoid duplication of large
blocks of flag setting between the async handler and register
functions.
Add new sysctl variables that describe zone support and paramters.
Add support for the new BIO_ZONE bio, and all of its subcommands:
DISK_ZONE_OPEN, DISK_ZONE_CLOSE, DISK_ZONE_FINISH, DISK_ZONE_RWP,
DISK_ZONE_REPORT_ZONES, and DISK_ZONE_GET_PARAMS.
sys/cam/scsi/scsi_all.c:
Add command descriptions for the ZBC IN/OUT commands.
Add descriptions for ZBC Host Managed devices.
Add a new function, scsi_ata_pass() to do ATA passthrough over
SCSI. This will eventually replace scsi_ata_pass_16() -- it
can create the 12, 16, and 32-byte variants of the ATA
PASS-THROUGH command, and supports setting all of the
registers defined as of SAT-4, Revision 5 (March 11, 2016).
Change scsi_ata_identify() to use scsi_ata_pass() instead of
scsi_ata_pass_16().
Add a new scsi_ata_read_log() function to facilitate reading
ATA logs via SCSI.
sys/cam/scsi/scsi_all.h:
Add the new ATA PASS-THROUGH(32) command CDB. Add extended and
variable CDB opcodes.
Add Zoned Block Device Characteristics VPD page.
Add ATA Return SCSI sense descriptor.
Add prototypes for scsi_ata_read_log() and scsi_ata_pass().
sys/cam/scsi/scsi_da.c:
Revamp the da(4) driver to support zoned devices.
Add five new probe states, four of which are needed for ATA
devices.
Add five new sysctl variables that describe zone support and
parameters.
The da(4) driver supports SCSI ZBC devices, as well as ATA ZAC
devices when they are attached via a SCSI to ATA Translation (SAT)
layer. Since ZBC -> ZAC translation is a new feature in the T10
SAT-4 spec, most SATA drives will be supported via ATA commands
sent via the SCSI ATA PASS-THROUGH command. The da(4) driver will
prefer the ZBC interface, if it is available, for performance
reasons, but will use the ATA PASS-THROUGH interface to the ZAC
command set if the SAT layer doesn't support translation yet.
As I mentioned above, ZBC command support is untested.
Add support for the new BIO_ZONE bio, and all of its subcommands:
DISK_ZONE_OPEN, DISK_ZONE_CLOSE, DISK_ZONE_FINISH, DISK_ZONE_RWP,
DISK_ZONE_REPORT_ZONES, and DISK_ZONE_GET_PARAMS.
Add scsi_zbc_in() and scsi_zbc_out() CCB building functions.
Add scsi_ata_zac_mgmt_out() and scsi_ata_zac_mgmt_in() CCB/CDB
building functions. Note that these have return values, unlike
almost all other CCB building functions in CAM. The reason is
that they can fail, depending upon the particular combination
of input parameters. The primary failure case is if the user
wants NCQ, but fails to specify additional CDB storage. NCQ
requires using the 32-byte version of the SCSI ATA PASS-THROUGH
command, and the current CAM CDB size is 16 bytes.
sys/cam/scsi/scsi_da.h:
Add ZBC IN and ZBC OUT CDBs and opcodes.
Add SCSI Report Zones data structures.
Add scsi_zbc_in(), scsi_zbc_out(), scsi_ata_zac_mgmt_out(), and
scsi_ata_zac_mgmt_in() prototypes.
sys/dev/ahci/ahci.c:
Fix SEND / RECEIVE FPDMA QUEUED in the ahci(4) driver.
ahci_setup_fis() previously set the top bits of the sector count
register in the FIS to 0 for FPDMA commands. This is okay for
read and write, because the PRIO field is in the only thing in
those bits, and we don't implement that further up the stack.
But, for SEND and RECEIVE FPDMA QUEUED, the subcommand is in that
byte, so it needs to be transmitted to the drive.
In ahci_setup_fis(), always set the the top 8 bits of the
sector count register. We need it in both the standard
and NCQ / FPDMA cases.
sys/geom/eli/g_eli.c:
Pass BIO_ZONE commands through the GELI class.
sys/geom/geom.h:
Add g_io_zonecmd() prototype.
sys/geom/geom_dev.c:
Add new DIOCZONECMD ioctl, which allows sending zone commands to
disks.
sys/geom/geom_disk.c:
Add support for BIO_ZONE commands.
sys/geom/geom_disk.h:
Add a new flag, DISKFLAG_CANZONE, that indicates that a given
GEOM disk client can handle BIO_ZONE commands.
sys/geom/geom_io.c:
Add a new function, g_io_zonecmd(), that handles execution of
BIO_ZONE commands.
Add permissions check for BIO_ZONE commands.
Add command decoding for BIO_ZONE commands.
sys/geom/geom_subr.c:
Add DDB command decoding for BIO_ZONE commands.
sys/kern/subr_devstat.c:
Record statistics for REPORT ZONES commands. Note that the
number of bytes transferred for REPORT ZONES won't quite match
what is received from the harware. This is because we're
necessarily counting bytes coming from the da(4) / ada(4) drivers,
which are using the disk_zone.h interface to communicate up
the stack. The structure sizes it uses are slightly different
than the SCSI and ATA structure sizes.
sys/sys/ata.h:
Add many bit and structure definitions for ZAC, NCQ, and EPC
command support.
sys/sys/bio.h:
Convert the bio_cmd field to a straight enumeration. This will
yield more space for additional commands in the future. After
change r297955 and other related changes, this is now possible.
Converting to an enumeration will also prevent use as a bitmask
in the future.
sys/sys/disk.h:
Define the DIOCZONECMD ioctl.
sys/sys/disk_zone.h:
Add a new API for managing zoned disks. This is very close to
the SCSI ZBC and ATA ZAC standards, but uses integers in native
byte order instead of big endian (SCSI) or little endian (ATA)
byte arrays.
This is intended to offer to the complete feature set of the ZBC
and ZAC disk management without requiring the application developer
to include SCSI or ATA headers. We also use one set of headers
for ioctl consumers and kernel bio-level consumers.
sys/sys/param.h:
Bump __FreeBSD_version for sys/bio.h command changes, and inclusion
of SMR support.
usr.sbin/Makefile:
Add the zonectl utility.
usr.sbin/diskinfo/diskinfo.c
Add disk zoning capability to the 'diskinfo -v' output.
usr.sbin/zonectl/Makefile:
Add zonectl makefile.
usr.sbin/zonectl/zonectl.8
zonectl(8) man page.
usr.sbin/zonectl/zonectl.c
The zonectl(8) utility. This allows managing SCSI or ATA zoned
disks via the disk_zone.h API. You can report zones, reset write
pointers, get parameters, etc.
Sponsored by: Spectra Logic
Differential Revision: https://reviews.freebsd.org/D6147
Reviewed by: wblock (documentation)
Currently, Application Processors (non-boot CPUs) are started by
MD code at SI_SUB_CPU, but they are kept waiting in a "pen" until
SI_SUB_SMP at which point they are released to run kernel threads.
SI_SUB_SMP is one of the last SYSINIT levels, so APs don't enter
the scheduler and start running threads until fairly late in the
boot.
This change moves SI_SUB_SMP up to just before software interrupt
threads are created allowing the APs to start executing kernel
threads much sooner (before any devices are probed). This allows
several initialization routines that need to perform initialization
on all CPUs to now perform that initialization in one step rather
than having to defer the AP initialization to a second SYSINIT run
at SI_SUB_SMP. It also permits all CPUs to be available for
handling interrupts before any devices are probed.
This last feature fixes a problem on with interrupt vector exhaustion.
Specifically, in the old model all device interrupts were routed
onto the boot CPU during boot. Later after the APs were released at
SI_SUB_SMP, interrupts were redistributed across all CPUs.
However, several drivers for multiqueue hardware allocate N interrupts
per CPU in the system. In a system with many CPUs, just a few drivers
doing this could exhaust the available pool of interrupt vectors on
the boot CPU as each driver was allocating N * mp_ncpu vectors on the
boot CPU. Now, drivers will allocate interrupts on their desired CPUs
during boot meaning that only N interrupts are allocated from the boot
CPU instead of N * mp_ncpu.
Some other bits of code can also be simplified as smp_started is
now true much earlier and will now always be true for these bits of
code. This removes the need to treat the single-CPU boot environment
as a special case.
As a transition aid, the new behavior is available under a new kernel
option (EARLY_AP_STARTUP). This will allow the option to be turned off
if need be during initial testing. I plan to enable this on x86 by
default in a followup commit in the next few days and to have all
platforms moved over before 11.0. Once the transition is complete,
the option will be removed along with the !EARLY_AP_STARTUP code.
These changes have only been tested on x86. Other platform maintainers
are encouraged to port their architectures over as well. The main
things to check for are any uses of smp_started in MD code that can be
simplified and SI_SUB_SMP SYSINITs in MD code that can be removed in
the EARLY_AP_STARTUP case (e.g. the interrupt shuffling).
PR: kern/199321
Reviewed by: markj, gnn, kib
Sponsored by: Netflix
In struct:gctl_req, nargs is unsigned.
In mirror:
g_mirror_syncreqs is unsigned.
In raid:
in struct:g_raid_volume, v_disks_count is unsigned.
In virstor:
in struct:g_virstor_softc, n_components is unsigned.
MFC after: 2 weeks
This value is u32 on disk, but assigned to an int in memory. After we do the
implicit conversion via assignment, check that the result is at least one[1]
(non-negative[2]).
1. The subsequent for-loop iterates from gpt_entries minus one, down, until
reaching zero. A negative or zero initial index results in undefined signed
integer overflow.
2. It is also used to index into arrays later.
In practice, we expected non-malicious disks to contain small positive values.
Reported by: Coverity
CID: 1223202
Sponsored by: EMC / Isilon Storage Division
rounddown2 tends to produce longer lines than the original code
and when the code has a high indentation level it was not really
advantageous to do the replacement.
This tries to strike a balance between readability using the macros
and flexibility of having the expressions, so not everything is
converted.
sys/geom/geom_disk.c:
disk_attr_changed(): Generate a devctl event of type GEOM:<attr> for
every call.
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D5952
This flag indicates that the user wishes to use the GELIBOOT feature to boot from a fully encrypted root file system.
Currently, GELIBOOT does not support key files, and in the future when it does, they will be loaded differently.
Due to the design of GELI, and the desire for secrecy, the GELI metadata does not know if key files are used or not, it just adds the key material (if any) to the HMAC before the optional passphrase, so there is no way to tell if a GELI partition requires key files or not.
Since the GELIBOOT code in boot2 and the loader does not support keys, they will now only attempt to attach if this flag is set. This will stop GELIBOOT from prompting for passwords to GELIs that it cannot decrypt, disrupting the boot process
PR: 208251
Reviewed by: ed, oshogbo, wblock
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D5867
For the !gsp case there some chance of returning an uninitialized
return value. Prevent that from happening by initializing the
error value.
CID: 1006421
and geom_uncompress(4):
1. mkuzip(8):
- Proper support for eliminating all-zero blocks when compressing an
image. This feature is already supported by the geom_uzip(4) module
and CLOOP format in general, so it's just a matter of making mkuzip(8)
match. It should be noted, however that this feature while it sounds
great, results in very slight improvement in the overall compression
ratio, since compressing default 16k all-zero block produces only 39
bytes compressed output block, which is 99.8% compression ratio. With
typical average compression ratio of amd64 binaries and data being
around 60-70% the difference between 99.8% and 100.0% is not that
great further diluted by the ratio of number of zero blocks in the
uncompressed image to the overall number of blocks being less than
0.5 (typically). However, this may be important from performance
standpoint, so that kernel are not spinning its wheels decompressing
those empty blocks every time this zero region is read. It could also
be important when you create huge image mostly filled with zero
blocks for testing purposes.
- New feature allowing to de-duplicate output image. It turns out that
if you twist CLOOP format a bit you can do that as well. And unlike
zero-blocks elimination, this gives a noticeable improvement in the
overall compression ratio, reducing output image by something like
3-4% on my test UFS2 3GB image consisting of full FreeBSD base system
plus some of the packages (openjdk, apache etc), about 2.3GB worth of
file data (800+MB compressed). The only caveat is that images created
with this feature "on" would not work on older versions of FeeBSDxi
kernel, hence it's turned off by default.
- provide options to control both features and document them in manual
page.
- merge in all relevant LZMA compression support from the mkulzma(8),
add new option to select between both.
- switch license from ad-hoc beerware into standard 2-clause BSD.
2. geom_uzip(4):
- implement support for de-duplicated images;
- optimize some code paths to handle "all-zero" blocks without reading
any compressed data;
- beef up manual page to explain that geom_uzip(4) is not limited only
to md(4) images. The compressed data can be written to the block
device and accessed directly via magic of GEOM(4) and devfs(4),
including to mount root fs from a compressed drive.
- convert debug log code from being compiled in conditionally into
being present all the time and provide two sysctls to turn it on or
off. Due to intended use of the module, it can be used in
environments where there may not be a luxury to put new kernel with
debug code enabled. Having those options handy allows debug issues
without as much problem by just having access to serial console or
network shell access to a box/appliance. The resulting additional
CPU cycles are just few int comparisons and branches, and those are
minuscule when compared to data decompression which is the main
feature of the module.
- hopefully improve robustness and resiliency of the geom_uzip(4) by
performing some of the data validation / range checking on the TOC
entries and rejecting to attach to an image if those checks fail.
- merge in all relevant LZMA decompression support from the
geom_uncompress(4), enable automatically when appropriate format is
indicated in the header.
- move compilation work into its own worker thread so that it does not
clog g_up. This allows multiple instances work in parallel utilizing
smp cores.
- document new knobs in the manual page.
Reviewed by: adrian
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D5333
