On an amd64-CURRENT machine with an i-node that refers to a block
number that is one too large will cause a core dump, due to writing
beyond the end of blockmap[] and corrupting the next heap block,
which happens to contain a struct inoinfo in inphash[]. Note that
valgrind catches the blockmap[] access.
Reported by: Robert Morris
PR: 271289
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
The previous change involved calling check_cgmagic() twice in a row
for the same CG in order to differentiate when the CG was already ok vs.
when the CG was rebuilt, but that doesn't work because the second call
(which was supposed to rebuild the CG) returns 0 (indicating that
the CG was not rebuilt) due to the prevfailcg check causing an early
failure return. Fix this by moving the rebuild part of check_cgmagic()
out into a separate function which is called by pass1() when it wants to
rebuild a CG.
Fixes: da86e7a20d
Reported by: pho
Discussed with: mckusick
Sponsored by: Netflix
Pass 1 of fsck_ffs checks the integrity of all the cylinder groups.
If any are found to have been corrupted it offers to rebuild them.
Pass 5 then makes a second pass over the cylinder groups to validate
their block and inode maps. Pass 5 assumes that the cylinder groups
are not corrupted and can segment fault if they are corrupted. Rather
than rerunning the corruption checks a second time in pass 5, this
fix keeps track whether any corrupt cylinder groups were found but not
fixed in pass 1 either due to running with the -n flag or by explicitly
answering `no' when asked whether to fix a corrupted cylinder group.
If any corrupted cylinder groups remain after pass 1, fsck_ffs will
decline to run pass 5. Instead it marks the filesystem as unclean
so that fsck_ffs will need to be run again before the filesystem can
be mounted.
This patch cleans up and documents the return value from check_cgmagic().
It also renames the variable / parameter "rebuildcg" to "rebuiltcg".
This parameter describes whether the cylinder group has been rebuilt
rather than whether it should be rebuilt.
Reported by: Chuck Silvers
Reviewed by: Chuck Silvers
MFC after: 1 week
Increment a reference count when returning a zero'ed out buffer
after a failed read.
Zero out a structure before using it.
Only dirty a buffer that has been modified.
Submitted by: Chuck Silvers
Sponsored by: Netflix
MFC after: 1 week
The algorithm for laying out new directories was devised in the 1980s
and markedly improved the performance of the filesystem. In those days
large disks had at most 100 cylinder groups and often as few as 10-20.
Modern multi-terrabyte disks have thousands of cylinder groups. The
original algorithm does not handle these large sizes well. This change
attempts to expand the scope of the original algorithm to work well
with these much larger disks while still retaining the properties
of the original algorithm for small disks.
The filesystem implementation is divided into policy routines and
implementation routines. The policy routines can be changed in any
way desired without risk of corrupting the filesystem. The policy
requests are handled by the implementation layer. If the policy
asks for an available resource, it is granted. But if it asks for
an already in-use resource, then the implementation will provide
an available one nearby the request. Thus it is impossible for a
policy to double allocate. This change is limited to the policy
implementation.
This change updates the ffs_dirpref() routine which is responsible
for selecting the cylinder group into which a new directory should
be placed. If we are near the root of the filesystem we aim to
spread them out as much as possible. As we descend deeper from the
root we cluster them closer together around their parent as we
expect them to be more closely interactive. Higher-level directories
like usr/src/sys and usr/src/bin should be separated while the
directories in these areas are more likely to be accessed together
so should be closer. And directories within commands or kernel
subsystems should be closer still.
We pick a range of cylinder groups around the cylinder group of the
directory in which we are being created. The size of the range for
our search is based on our depth from the root of our filesystem.
We then probe that range based on how many directories are already
present. The first new directory is at 1/2 (middle) of the range;
the second is in the first 1/4 of the range, then at 3/4, 1/8, 3/8,
5/8, 7/8, 1/16, 3/16, 5/16, etc.
It is desirable to store the depth of a directory in its on-disk
inode so that it is available when we need it. We add a new field
di_dirdepth to track the depth of each directory. Because there are
few spare fields left in the inode, we choose to share an existing
field in the inode rather than having one of our own. Specifically
we create a union with the di_freelink field. The di_freelink field
is used to track inodes that have been unlinked but remain referenced.
It is not needed until a rmdir(2) operation has been done on a
directory. At that point, the directory has no contents and even
if it is kept active as a current directory is no longer able to
have any new directories or files created in it. Thus the use of
di_dirdepth and di_freelink will never coincide.
Reported by: Timo Voelker
Reviewed by: kib
Tested by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39246
If a directory entry has an illegal inode number (less than zero
or greater than the last inode in the filesystem) the entry is removed.
If a directory '.' or '..' entry had an illegal inode number they
were being removed. Since fsck_ffs knows what the correct value is
for these two entries fix them rather deleting them.
Add much more extensive cylinder group checks and use them to be
more careful about rebuilding a cylinder group.
Check for out-of-range block numbers before trying to free them.
When a directory is deleted also remove its cache entry created
in pass1 so that later passes do not try to operate on a deleted
directory.
Check for ctime(3) returning NULL before trying to use its return.
When freeing a directory inode, do not try to interpret it as a
directory.
Reserve space in the inostatlist to have room to allocate a
lost+found directory.
If an invalid block number is found past the end of an inode simply
remove it rather than clearing and removing the inode.
Modernize the inoinfo structure to use queue(3) LIST rather than a
handrolled linked list implementation.
Reported by: Bob Prohaska, John-Mark Gurney, and Mark Millard
Tested by: Peter Holm
Reviewed by: Peter Holm
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D38668
Use it instead of defining IN_RTLD by base sources that want P_OSREL_
defines in userspace, but are not rtld.
This allows to remove abuse of IN_RTLD from userspace.
Reviewed by: dchagin, markj, imp
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D38585
The mntopts(3) functions support operations associated with a mount
point. The main purpose of this commit is to document the mntopts(3)
functions that now appear in 18 utilities in the base system. See
mntopts(3) for the documentation details.
The getmntopts() function appeared in 4.4BSD. The build_iovec(),
build_iovec_argf(), free_iovec(), checkpath(), and rmslashes()
functions were added with nmount(8) in FreeBSD 5.0. The getmntpoint()
and chkdoreload() functions are being added in this commit.
These functions should be in a library but for historic reasons are
in a file in the sources for the mount(8) program. Thus, to access
them the following lines need to be added to the Makefile of the
program wanting to use them:
SRCS+= getmntopts.c
MOUNT= ${SRCTOP}/sbin/mount
CFLAGS+= -I${MOUNT}
.PATH: ${MOUNT}
Once these changes have been MFC'ed to 13 they may be made into
a library.
Reviewed by: kib, gbe
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37907
The kernel handles the managment of UFS/FFS snapshots. Since UFS/FFS
updates filesystem data (rather than always writing changes to new
locations like ZFS), the kernel must check every filesystem write
to see if the block being written is part of a snapshot. If it is
part of a snapshot, then the kernel must make a copy of the old
block value into a newly allocated block for the snapshot before
allowing the write to be done. Similarly, if a block is being freed,
the kernel must check to see if it is part of a snapshot and let
the snapshot claim the block rather than freeing it for future use.
When a snapshot is freed, its blocks need to be offered to older
snapshots and freed only if no older snapshots wish to claim them.
When snapshots were added to UFS/FFS they were integrated into soft
updates and just a small part of the management of snapshots needed
to be added to fsck_ffs(8) as soft updates minimized the set of
snapshot changes that might need correction. When journaling was
added to soft updates a much more complete knowledge of snapshots
needed to be added to fsck_ffs(8) for it to be able to properly
handle the filesystem changes that a journal rollback needs to do
(specifically the freeing and allocation of blocks). Since this
functionality was unavailable, the use of snapshots was disabled
when running with journaled soft updates.
This set of changes imports the kernel code for the management of
snapshots to fsck_ffs(8). With this code in place it will become
possible to enable snapshots when running with journalled soft
updates. The most immediate benefit will be the ability to use
snapshots to take consistent filesystem dumps on live filesystems.
Future work will be done to update fsck_ffs(8) to be able to use
snapshots to run in background on live filesystems running with
journaled soft updates.
Reviewed by: kib
Tested by: Peter Holm
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36491
It's not really useful in a jail or in a mdroot or even if a users
wants to do a full zfs machine.
Reviewed by: mckusick
Differential Revision: https://reviews.freebsd.org/D36227
When something was found wrong with an inode the error message
was always "UNKNOWN FILE TYPE". This error is now used only when
the file type field is wrong. Other errors have their own messages:
"BAD FILE SIZE", "NEGATIVE FILE SIZE", "BAD SPECIAL-FILE RDEV",
"INVALID DIRECT BLOCK", and "INVALID INDIRECT BLOCK".
More complete information about the inode is also provided.
Sponsored by: The FreeBSD Foundation
If the root directory exists but has a bad block number Pass1 will
accept it and setup an inoinfo structure for it. When Pass2 runs
and cannot read the root inode's content because of a bad (or
duplicate) block number, it removes the bad root inode and replaces
it. As part of creating the replacement root inode, it creates an
inoinfo entry for it. But Pass2 did delete the inoinfo entry that
Pass1 had set up for the root inode so ended up with two inoinfo
structures for it. The final step of Pass2 checks that all the ".."
entries are correct adding them if they are missing which resulted
in a second ".." entry being added to the root directory which
definitely did not go over well in the kernel name cache!
Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation
Like the detailed diagnostics produced when a bad superblock
is read, provide similar detailed diagnostics when bad
cylinder groups are read.
Reported by: Peter Holm
Tested by: Peter Holm
Sponsored by: The FreeBSD Foundation
The fsck_ffs(8) utility has two subsystems for reading and writing
inodes. The getnextinode() interface is used in Pass 1 (and Pass
1b if needed) to sequentially walk through all the inodes in the
filesystem. The ginode() interface is used to read and write
individual inodes. Pass 1 uses a mix of both interfaces. This
change ensures that ginode() returns a pointer to the inode in the
cache maintained by getnextinode() when that interface holds the
requested inode so that all modifications to the inode are made in
a single place and are all written to the disk together.
Reported by: Peter Holm
Tested by: Peter Holm
Sponsored by: The FreeBSD Foundation
Historically fsck_ffs(8) would only use alternate superblocks when
running in manual mode. When the standard superblock fails, it now
tries to find and use a backup superblocks even when running in `preen'
mode. If an alternate superblock is found and the filesystem is
successfully cleaned up using it, write the alternate superblock
back to the standard superblock so that the filesystem can be
subsequently mounted and used.
Reported by: Peter Holm
Tested by: Peter Holm
Sponsored by: The FreeBSD Foundation
The UFS filesystem expects to find '.' and '..' as the first two entries
in a directory. The kernel's UFS name cache can become quite confused
when these two entries are not present as the first two entries.
Prior to this change, when the fsck_ffs(8) utility detected that
'.' and/or '..' were missing, it would report them, but only offered
to replace them if the space at the beginning of the directory was
available. Otherwise it was left to the system administrator to
move the offending file(s) out of the way and then rerun fsck_ffs(8)
to create the '.' and '..' entries.
With this change, fsck_ffs(8) will always be able to create the '.'
and/or '..' entries. It moves any files in the way elsewhere in the
directory block. If there is no room in the directory block to which
to move them, they are placed in the lost+found directory.
Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation
When either searching for backup UFS superblocks or when explicitly asked
to use one with the -b option, report the reason for failure if it cannot
be used.
Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation
Allocation or I/O failures in fsck_ffs(8) could cause segment
faults because of missing checks or not-yet-initialized data
structures. Correct these issues.
Reported by: Peter Holm
Sponsored by: The FreeBSD Foundation
into ffs_sbsearch() to allow use by other parts of the system.
Historically only fsck_ffs(8), the UFS filesystem checker, had code
to track down and use alternate UFS superblocks. Since fsdb(8) used
much of the fsck_ffs(8) implementation it had some ability to track
down alternate superblocks.
This change extracts the code to track down alternate superblocks
from fsck_ffs(8) and puts it into a new function ffs_sbsearch() in
sys/ufs/ffs/ffs_subr.c. Like ffs_sbget() and ffs_sbput() also found
in ffs_subr.c, these functions can be used directly by the kernel
subsystems. Additionally they are exported to the UFS library,
libufs(8) so that they can be used by user-level programs. The new
functions added to libufs(8) are sbfind(3) that is an alternative
to sbread(3) and sbsearch(3) that is an alternative to sbget(3).
See their manual pages for further details.
The utilities that have been changed to search for superblocks are
dumpfs(8), fsdb(8), ffsinfo(8), and fsck_ffs(8). Also, the prtblknos(8)
tool found in tools/diag/prtblknos searches for superblocks.
The UFS specific mount code uses the superblock search interface
when mounting the root filesystem and when the administrator doing
a mount(8) command specifies the force flag (-f). The standalone UFS
boot code (found in stand/libsa/ufs.c) uses the superblock search
code in the hope of being able to get the system up and running so
that fsck_ffs(8) can be used to get the filesystem cleaned up.
The following utilities have not been changed to search for
superblocks: clri(8), tunefs(8), snapinfo(8), fstyp(8), quot(8),
dump(8), fsirand(8), growfs(8), quotacheck(8), gjournal(8), and
glabel(8). When these utilities fail, they do report the cause of
the failure. The one exception is the tasting code used to try and
figure what a given disk contains. The tasting code will remain
silent so as not to put out a slew of messages as it trying to taste
every new mass storage device that shows up.
Reviewed by: kib
Reviewed by: Warner Losh
Tested by: Peter Holm
Differential Revision: https://reviews.freebsd.org/D36053
Sponsored by: The FreeBSD Foundation
The fsck_ffs(8) utility made sanity checks of critical superblock
fields by comparing the values of those fields in the standard
superblock againt the values of those fields in the last alternate
superblock. The code for validating a superblock now cover all the
checked fields as well as many more. Further the checks done are
far more comprehensive. So we now drop the alternate superblock
checks as they no longer provide value. Dropping these checks also
eliminates the need to read the alternate superblock.
Rather than trying to shoehorn flags into the requested superblock
address, create a separate flags parameter to the ffs_sbget()
function in sys/ufs/ffs/ffs_subr.c. The ffs_sbget() function is
used both in the kernel and in user-level utilities through export
to the sbget() function in the libufs(3) library (see sbget(3)
for details). The kernel uses ffs_sbget() when mounting UFS
filesystems, in the glabel(8) and gjournal(8) GEOM utilities,
and in the standalone library used when booting the system
from a UFS root filesystem.
The ffs_sbget() function reads the superblock located at the byte
offset specified by its sblockloc parameter. The value UFS_STDSB
may be specified for sblockloc to request that the standard
location for the superblock be read.
The two existing options are now flags:
UFS_NOHASHFAIL will note if the check hash is wrong but will still
return the superblock. This is used by the bootstrap code to
give the system a chance to come up so that fsck can be run to
correct the problem.
UFS_NOMSG indicates that superblock inconsistency error messages
should not be printed. It is used by programs like fsck that
want to print their own error message and programs like glabel(8)
that just want to know if a UFS filesystem exists on a partition.
One additional flag is added:
UFS_NOCSUM causes only the superblock itself to be returned, but does
not read in any auxiliary data structures like the cylinder group
summary information. It is used by clients like glabel(8) that
just want to check for possible filesystem types. Using UFS_NOCSUM
skips the superblock checks for csum data which allows superblocks
that have corrupted csum data to be read and used.
The validate_sblock() function checks that the superblock has not
been corrupted in a way that can crash or hang the system. Unless
the UFS_NOMSG flag is specified, it will print out any errors that
it finds. Prior to this commit, validate_sblock() returned as soon
as it found an inconsistency so would print at most one message.
It now does all its checks so when UFS_NOMSG has not been specified
will print out everything that it finds inconsistent.
Sponsored by: The FreeBSD Foundation
for missing devices.
The fsck_ffs(8) utility uses its internal function openfilesys()
when opening a disk to be checked. This change avoids the use
of pfatal() in openfilesys() which always exits with failure (exit
value 8) so that the caller can choose the correct exit value.
In the case of a non-existent device it should exit with value 3
which allows the startup system to wait for drives (such as those
attached by USB) to come online.
Reported by: karels
Tested by: karels
PR: 262580
MFC after: 3 days
The cleanup of fsck_ffs(8) in commit c0bfa109b9 broke fsdb(8).
This commit adds the one-line update needed in fsdb(8) to make it
work with the new fsck_ffs(8) structure.
Reported by: Chuck Silvers
Tested by: Chuck Silvers
MFC after: 3 days
Normally fsck_ffs never does reads or writes that are not aligned
to the size of one of the checked filesystems fragments. The one
exception is when it finds that it needs to write the superblock
recovery information. Here it will write with the alignment reported
by the underlying disk as its sector size as reported by an
ioctl(diskfd, DIOCGSECTORSIZE, &secsize).
Modern disks have a sector size of 4096, but for backward compatibility
with older disks will report that they have a sector size of 512.
When presented with a 512 byte write, they have to read the associated
4096 byte sector, replace the 512 bytes to be written, and write
the updated 4096 byte sector back to the disk. Unfortunately, some
disks report that they have 512 sectors, but fail writes that are not
aligned to 4096 boundaries and are a multiple of 4096 bytes in size.
This commit updates fsck_ffs(8) so that it uses the filesystem fragment
size as the smallest size and alignment for doing writes rather than
the disk's reported sector size.
Reported by: Andriy Gapon
MFC after: 1 week
Part of the problem was that fsck_ffs would read the superblock
multiple times complaining and repairing the superblock check hash
each time and then at the end failing to write out the superblock
with the corrected check hash. This fix reads the superblock just
once and if the check hash is corrected ensures that the fixed
superblock gets written.
Tested by: Peter Holm
PR: 245916
MFC after: 1 week
Sponsored by: Netflix
The ino_blkatoff() and indir_blkatoff() functions failed to release
the buffers holding second and third level indirect blocks. This
commit ensures that these buffers are now properly released.
MFC after: 1 week
Sponsored by: Netflix
When fsck_ffs is running in preen mode and finds a zero-length directory,
it deletes that directory. In doing this operation, it unnecessary set
its internal flag saying that fsck_ffs needed to be rerun. This patch
deletes the rerun request for this case.
Reported by: Mark Johnson
PR: 246962
MFC after: 1 week
Sponsored by: Netflix
Background checks are only allowed for mounted filesystems - don't try
to open the device for writing when performing a background check.
While here, remove a debugging printf that's commented out.
PR: 256746
Fixes: 5cc52631b3
Reviewed by: mckusick
MFC After: 1 week
Differential Revision: https://reviews.freebsd.org/D30880
These were all incorrectly labeled as 2-clause BSD licenses by a
semi-automated process, when in fact they are 3-clause.
Discussed with: pfg, imp
MFC after: 2 weeks
Sponsored by: Axcient
