The index() and rindex() functions were marked LEGACY in the 2001
revision of POSIX and were subsequently removed from the 2008 revision.
The strchr() and strrchr() functions are part of the C standard.
This makes the source code a lot more consistent, as most of these C
files also call into other str*() routines. In fact, about a dozen
already perform strchr() calls.
The value of namlen is copied from on-disk d_namlen, which is a 8-bit
unsigned integer which can never exceed MAXNAMLEN (255) so the test is
always true. Moreover, UFS does not allow d_namelen being zero.
Change namlen from u_int to u_int8_t, and replace the unneeded test
with a useful test.
PR: bin/160339
Submitted by: Eugene Grosbein <eugen grosbein.pp.ru>
MFC after: 2 weeks
Approved by: re (kib)
to resolve errors which can cause corruption on recovery with the old
synchronous mechanism.
- Append partial truncation freework structures to indirdeps while
truncation is proceeding. These prevent new block pointers from
becoming valid until truncation completes and serialize truncations.
- On completion of a partial truncate journal work waits for zeroed
pointers to hit indirects.
- softdep_journal_freeblocks() handles last frag allocation and last
block zeroing.
- vtruncbuf/ffs_page_remove moved into softdep_*_freeblocks() so it
is only implemented in one place.
- Block allocation failure handling moved up one level so it does not
proceed with buf locks held. This permits us to do more extensive
reclaims when filesystem space is exhausted.
- softdep_sync_metadata() is broken into two parts, the first executes
once at the start of ffs_syncvnode() and flushes truncations and
inode dependencies. The second is called on each locked buf. This
eliminates excessive looping and rollbacks.
- Improve the mechanism in process_worklist_item() that handles
acquiring vnode locks for handle_workitem_remove() so that it works
more generally and does not loop excessively over the same worklist
items on each call.
- Don't corrupt directories by zeroing the tail in fsck. This is only
done for regular files.
- Push a fsync complete record for files that need it so the checker
knows a truncation in the journal is no longer valid.
Discussed with: mckusick, kib (ffs_pages_remove and ffs_truncate parts)
Tested by: pho
that was built before ffs grew support for TRIM, your filesystem will have
plenty of free blocks that the flash chip doesn't know are free, so it
can't take advantage of them for wear leveling. Once you've upgraded your
kernel, you enable TRIM on the filesystem (tunefs -t enable), then run
fsck_ffs -E on it before mounting it.
I tested this patch by half-filling an mdconfig'ed filesystem image,
running fsck_ffs -E on it, then verifying that the contents were not
damaged by comparing them to a pristine copy using rsync's checksum
functionality. There is no reliable way to test it on real hardware.
Many thanks to mckusick@, who provided the tricky parts of this patch and
reviewed the final version.
Reviewed by: mckusick@
MFC after: 3 weeks
Due to UFS insistence to pretend that device sector size is 512 bytes,
sector size is obtained from ioctl(DIOCGSECTORSIZE) for real devices,
and from the label otherwise. The file images without label have to
be made with 512 sector size.
In collaboration with: pho
Reviewed by: jeff
Tested by: bz, pho
include sys/time.h instead of time.h. This include is incorrect as
per the manpages for the APIs and the POSIX definitions. This commit
replaces sys/time.h where necessary with time.h.
The commit also includes some minor style(9) header fixup in newfs.
This commit is part of a larger effort by Garrett Cooper started in
//depot/user/gcooper/posix-conformance-work/ -- to make FreeBSD more
POSIX compliant.
Submitted by: Garrett Cooper yanegomi at gmail dot com
directory truncation to proceed before the link has been cleared. This
is accomplished by detecting a directory with no . or .. links and
clearing the named directory entry in the parent.
- Add a new function ino_remref() which handles the details of removing
a reference to an inode as a result of a lost directory. There were
some minor errors in various subcases of this routine.
- Use err/errx only when the case is really fatal. For other
cases, fall back to full fsck instead of quiting fsck.
- Plug a memory leak.
- Avoid divide by zero when printing summary.
- Output "FILE SYSTEM IS MARKED CLEAN" when a successful
journal recovering is done.
- When -f is specified, do full fsck instead of journal recovery.
brings in support for an optional intent log which eliminates the need
for background fsck on unclean shutdown.
Sponsored by: iXsystems, Yahoo!, and Juniper.
With help from: McKusick and Peter Holm
in background mode to correct expected inconsistencies that arise
during directory rename (see immediately previous update to this
file for details). If run on a kernel without the new functionality,
background fsck will simply ignore these inconsistencies rather
than fail.
Reported by: jeff
states. First its new name will be created causing it to have two
names (from possibly different parents). Next, if it has different
parents, its value of ".." will be changed from pointing to the old
parent to pointing to the new parent. Concurrently, its old name
will be removed bringing it back into a consistent state. When fsck
encounters an extra name for a directory, it offers to remove the
"extraneous hard link"; when it finds that the names have been
changed but the update to ".." has not happened, it offers to rewrite
".." to point at the correct parent. Both of these changes were
considered unexpected so would cause fsck in preen mode or fsck in
background mode to fail with the need to run fsck manually to fix
these problems.
This update changes these errors to be expected so that in preen
mode fsck will simply fix these transitional errors. For now,
background fsck will note these errors, but will need additional
kernel support to fix them, so will simply ignore them rather than
fail. A future update will allow background fsck to fix these
problems.
Reported by: jeff
cylinder group of a UFS1 filesystem as bad. The error was in the check
and not in the cylinder group itself. So even though fsck fixed the
cylinder group correctly, it was still endlessly reported as bad.
PR: 141992
MFC after: 2 weeks
Reported by: Dan Strick
Fix some wrong usages.
Note: this does not affect generated binaries as this argument is not used.
PR: 137213
Submitted by: Eygene Ryabinkin (initial version)
MFC after: 1 month
robust. With these changes fsck is now able to detect and reliably
rebuild corrupted cylinder group maps. The -D option is no longer
necessary as it has been replaced by a prompt asking whether the
corrupted cylinder group should be rebuilt and doing so when requested.
These actions are only offered and taken when running fsck in manual
mode. Corrupted cylinder groups found during preen mode cause the fsck
to fail.
Add the -r option to free up excess unused inodes. Decreasing the
number of preallocated inodes reduces the running time of future
runs of fsck and frees up space that can allocated to files. The -r
option is ignored when running in preen mode.
Reviewed by: Xin LI <delphij@>
Sponsored by: Rsync.net
background fsck on the same file system might then print negative
numbers for reclaimed directories/files/fragments.
Address the issue in a limited degree, by using old summary data for
cg when bgfsck is performed.
Submitted by: tegge
MFC after: 1 week
systems less than 1 TB, due to using 32-bits integers for file system block
numbers. This also causes incorrect error reporting for foreground fsck.
Convert it to use ufs2_daddr_t for block numbers.
PR: kern/127951
Submitted by: tegge
MFC after: 1 week
userspace to kernel via nmount(), pass in the strings
"update", "snapshot", "reload".
We want to move away from passing MNT_ flags from userspace -> kernel
via nmount(), and instead favor passing the string options.
catastrophic recovery. Currently, this mode only validates whether a
cylindergroup has good signature data, and prompts the user to decide
whether to clear it as a whole.
This mode is useful when there is data damage on a disk and you are
working on copy of the original disk, as fsck_ffs(8) tends to abnormally
exit in such case, as a last resort to recover data from the disk.
doing the MNT_RELOAD, pass in "ro" and "update"
string mount options to nmount() instead of MNT_RDONLY and MNT_UPDATE flags.
Due to the complexity of the mount parsing code especially
with respect to the root file system, passing in MNT_RDONLY and MNT_UPDATE
flags would do weird things and would cause fsck to convert the root
file system from a read-only mount to read-write.
To test:
- boot into single user mode
- show mounted file systems with: mount
- root file system should be mounted read-only
- fsck /
- show mounted file systems with: mount
- root file system should still be mounted read-only
PR: 120319
MFC after: 1 month
Reported by: yar
number read from cylinder group. Chances that we read a smarshed
cylinder group, and we can not 100% trust information it has
supplied. fsck_ffs(8) will crash otherwise for some cases.
processing the information. chk1 is more prone to crash when insane
information is provided by the on-disk inode, and does not even work
if the inode is being smarshed badly.
whether fs_bsize is larger than MINBSIZE, which is larger than the
value that is used to compared with fs_bsize, the sizeof fs, so the
check followed, will be always true.
By inspecting the code and some old commit log, I believe that the
check must be that *fs_sbsize* is larger than sizeof fs. We round
up the size to nearest dev_bsize, as the smallest accepted fs_sbsize,
personally, I think this can be even changed to equal, because this
number is mostly an invariant in file systems.
With this check, fsck_ffs(8) will be more picky and has better
chance rejecting bad first superblock rather than referring to bad
value it supplied, thus gives better chance for it to check the
filesystem carefully.
read-only, so we can't simply exit right after calling gjournal_check(),
instead we need to ask about super block reload.
Submitted by: Niki Denev <niki@totalterror.net>
PR: misc/113889
Approved by: re (kensmith)
and -p flag was given perform fast file system checking (bascially only
garbage collecting of orphaned objects).
Rename bread() to blread() and bwrite() to blwrite() as we now link to
the libufs library, which also implement functions with that names.
Sponsored by: home.pl
initializing the sysctl mibs data before actually using them.
The original patchset (which is the actual version that is running
on my testboxes) have checked whether all of these sysctls and
refuses to do background fsck if we don't have them. Kirk has
pointed out that refusing running fsck on old kernels is pointless,
as old kernels will recompute the summary at mount time, so I
have removed these checks.
Unfortunatelly, as the checks will initialize the mib values of
those sysctl's, and which are vital for the runtime summary
adjustment to work, we can not simply remove the check, which
will lead to problem when running background fsck over a dirty
volume. Add these checks in a different way: give a warning rather
than refusing to work, and complain if the functionality is not
available when adjustments are necessary.
Noticed by: A power failure at my lab
Pointy hat: me
MFC After: 3 days
very slow process, especially for large file systems that is just
recovered from a crash.
Since the summary is already re-sync'ed every 30 second, we will
not lag behind too much after a crash. With this consideration
in mind, it is more reasonable to transfer the responsibility to
background fsck, to reduce the delay after a crash.
Add a new sysctl variable, vfs.ffs.compute_summary_at_mount, to
control this behavior. When set to nonzero, we will get the
"old" behavior, that the summary is computed immediately at mount
time.
Add five new sysctl variables to adjust ndir, nbfree, nifree,
nffree and numclusters respectively. Teach fsck_ffs about these
API, however, intentionally not to check the existence, since
kernels without these sysctls must have recomputed the summary
and hence no adjustments are necessary.
This change has eliminated the usual tens of minutes of delay of
mounting large dirty volumes.
Reviewed by: mckusick
MFC After: 1 week
count of zero and instead encode this information in the inode state.
Pass 4 performed a linear search of this list for each inode in
the file system, which performs poorly if the list is long.
Reviewed by: sam & keramida (an earlier version of the patch), mckusick
MFC after: 1 month
has only been partly initialized via newfs(8) so that it applies to both
UFS1 and UFS2.
Submitted by: "Xin LI" delphij at frontfree dot net
MFC: maybe?
shuffles the timing and sleep calls in bgfsck from:
sleep timer_on io timer_off io io io io io io io
to
sleep io io io io io io io timer_on io timer_off
The original method basically guaranteed that the timed I/O included a
disk seek every time, which made bgfsck sleep for much longer than
necessary.
Submitted by: Dan Nelson
Reviewed by: kirk
original intention of the less restrictive permissions was to allow
users to move or delete recovered files that they own. However, it
is better to not create world-writable directories by default; the
administrator can always pre-create lost+found if different permissions
are desired.
Reviewed by: mckusick
filesystem that is checked in background. Create the snapshot in this
directory rather than in the root. There are two benefits:
1) For terabyte-sized filesystems, the snapshot may require many
minutes to build. Although the filesystem will not be suspended
during most of the snapshot build, the snapshot file itself is
locked during the entire snapshot build period. Thus, if it is
accessed during the period that it is being built, the process
trying to access it will block holding its containing directory
locked. If the snapshot is in the root, the root will lock and
the system will come to a halt until the snapshot finishes. By
putting the snapshot in a subdirectory, it is out of the likely
path of any process traversing through the root and hence much
less likely to cause a lock race to the root.
2) The dump program is usually run by a non-root user running with
operator group privilege. Such a user is typically not permitted
to create files in the root of a filesystem. By having a directory
in group operator with group write access available, such a user
will be able to create a snapshot there. Having the dump program
create its snapshot in a subdirectory below the root will benefit
from point (1) as well.
Sponsored by: DARPA & NAI Labs.
bandwidth for other processes. Since the sleeping is done from
userland, this avoids the locking issues that affected the kernel
version.
The algorithm used here is to measure a moving average of the times
taken by a sample of read operations and then delay 1 in 8 reads
by 16 times the measured average. This should correspond to a factor
of 3 slowdown, but in practice the factor is larger (3.5 to 4) due
to hz rounding effects.
Reviewed by: mckusick
Approved by: re
trying to use them. Set a minimum value for numdirs when using an
alternate superblock to avoid spurious numdirs == 0 error. Calculate
new fields when using an alternate superblock from a UFS1 filesystem
to avoid segment faulting.
Sponsored by: DARPA & NAI Labs.
the old 8-bit fs_old_flags to the new location the first time that the
filesystem is mounted by a new kernel. One of the unused flags in
fs_old_flags is used to indicate that the flags have been moved.
Leave the fs_old_flags word intact so that it will work properly if
used on an old kernel.
Change the fs_sblockloc superblock location field to be in units
of bytes instead of in units of filesystem fragments. The old units
did not work properly when the fragment size exceeeded the superblock
size (8192). Update old fs_sblockloc values at the same time that
the flags are moved.
Suggested by: BOUWSMA Barry <freebsd-misuser@netscum.dyndns.dk>
Sponsored by: DARPA & NAI Labs.
It seems a common corruption to have them -ve (I've seen it several times)
and if fsck doesn't fix it, it leads to a kernel pagefault.
Reviewd by: kirk
Submitted by: Eric Jacobs <eaja@erols.com> and me independently.
MFC in: 2 days
PR: bin/40967
Approved by: re
fsck_ffs did not need it, but quotacheck did include it from fsck_ffs.
A repocopy has now moved the fsck_ffs/preen.c file to quotacheck/preen.c
quotacheck and fsck should probably use the same checkfstab() function
and it should possibly live in libufs.
Trouble is: they have diverged in the meantime.
At least now fsck_ffs is not in the equation anymore.
Sponsored by: DARPA & NAI Labs.
UFS2 commit.
These bits in essence made any instance of "softupdates expected
corrution", (ie blocks marked allocated but not referenced by an
inode etc) result in a exit value for fsck_ffs of 2.
2 is part of the magic and appearantly undocumented protocol between
fsck_FOO and fsck and means "dump into single user mode ASAP.
Sponsored by: DARPA & NAI Labs.
imposed by the filesystem structure itself remains. With 16k blocks,
the maximum file size is now just over 128TB.
For now, the UFS1 file size limit is left unchanged so as to remain
consistent with RELENG_4, but it too could be removed in the future.
Reviewed by: mckusick
filesystem expands the inode to 256 bytes to make space for 64-bit
block pointers. It also adds a file-creation time field, an ability
to use jumbo blocks per inode to allow extent like pointer density,
and space for extended attributes (up to twice the filesystem block
size worth of attributes, e.g., on a 16K filesystem, there is space
for 32K of attributes). UFS2 fully supports and runs existing UFS1
filesystems. New filesystems built using newfs can be built in either
UFS1 or UFS2 format using the -O option. In this commit UFS1 is
the default format, so if you want to build UFS2 format filesystems,
you must specify -O 2. This default will be changed to UFS2 when
UFS2 proves itself to be stable. In this commit the boot code for
reading UFS2 filesystems is not compiled (see /sys/boot/common/ufsread.c)
as there is insufficient space in the boot block. Once the size of the
boot block is increased, this code can be defined.
Things to note: the definition of SBSIZE has changed to SBLOCKSIZE.
The header file <ufs/ufs/dinode.h> must be included before
<ufs/ffs/fs.h> so as to get the definitions of ufs2_daddr_t and
ufs_lbn_t.
Still TODO:
Verify that the first level bootstraps work for all the architectures.
Convert the utility ffsinfo to understand UFS2 and test growfs.
Add support for the extended attribute storage. Update soft updates
to ensure integrity of extended attribute storage. Switch the
current extended attribute interfaces to use the extended attribute
storage. Add the extent like functionality (framework is there,
but is currently never used).
Sponsored by: DARPA & NAI Labs.
Reviewed by: Poul-Henning Kamp <phk@freebsd.org>
that might have changed, then did a byte-by-byte comparison with
the alternate. If any unused fields got used, they had to be added
to the exception list. Such changes caused too many false alarms.
So, I have changed the comparison algorithm to compare a selected
set of fields that are not expected to change. This new algorithm
causes far fewer false hits and still does a good job of detecting
problems when they have really occurred. In particular, this change
should ease the transition to kernels supporting UFS2 which make
some significant changes to the superblock.
Sponsored by: DARPA, NAI Labs
It does not help modern compilers, and some may take some hit from it.
(I also found several functions that listed *every* of its 10 local vars with
"register" -- just how many free registers do people think machines have?)
These were mainly missing casts or wrong format strings in printf
statements, but there were also missing includes, unused variables,
functions and arguments.
The choice of `long' vs `int' still seems almost random in a lot
of places though.
directory is encountered. This includes the full path of the
directory that will be removed if the user answers "y" to the
"REMOVE?" question.
PR: bin/226851
Submitted by: KOIE Hide <hide@koie.org>
MFC after: 1 week
when comparing with the alternate superblock. These fields are used
for temporary in-core information only. This should fix the "VALUES
IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST ALTERNATE" error from
fsck_ffs that has been seen a lot recently.
filesystem needs foreground checking (usually at boot time) or
can defer to background checking (after the system is up and running).
See the manual page, fsck_ffs(8), for details on the -F and -B options.
These options are primarily intended for use by the fsck front end.
All output is directed to stdout so that the output is coherent
when redirected to a file or a pipe. Unify the code with the fsck
front end that allows either a device or a mount point to be
specified as the argument to be checked.
1) Set the FS_NEEDSFSCK flag when unexpected problems are encountered.
2) Clear the FS_NEEDSFSCK flag after a successful foreground cleanup.
3) Refuse to run in background when the FS_NEEDSFSCK flag is set.
4) Avoid taking and removing a snapshot when the filesystem is already clean.
5) Properly implement the force cleaning (-f) flag when in preen mode.
Note that you need to have revision 1.21 (date: 2001/04/14 05:26:28) of
fs.h installed in <ufs/ffs/fs.h> defining FS_NEEDSFSCK for this to compile.
affect current systems until fsck is modified to use these new
facilities. To try out this change, set the fsck passno to zero
in /etc/fstab to cause the filesystem to be mounted without running
fsck, then run `fsck_ffs -p -B <filesystem>' after the system has
been brought up multiuser to run a background cleanup on <filesystem>.
Note that the <filesystem> in question must have soft updates enabled.
field, so it was possible for a filesystem marked clean by fsck_ffs
to cause kernel crashes later when mounted. This could occur when
fsck_ffs was used to repair a badly corrupted filesystem.
As pointed out by bde, it is not sufficient to restrict di_size to
just the superblock fs_maxfilesize limit. The use of 32-bit logical
block numbers (both in fsck and the kernel) induces another file
size limit which is usually lower than fs_maxfilesize. Also, the
old 4.3BSD filesystem does not have fs_maxfilesize initialised.
Following this change, fsck_ffs will enforce exactly the same
file size limits as are used by the kernel.
PR: kern/15065
Discussed with: bde
Reviewed by: bde, mckusick
in-core pointers to summary information. An array in this region
(fs_csp) could overflow on filesystems with a very large number of
cylinder groups (~16000 on i386 with 8k blocks). When this happens,
other fields in the superblock get corrupted, and fsck refuses to
check the filesystem.
Solve this problem by replacing the fs_csp array in 'struct fs'
with a single pointer, and add padding to keep the length of the
128-byte region fixed. Update the kernel and userland utilities
to use just this single pointer.
With this change, the kernel no longer makes use of the superblock
fields 'fs_csshift' and 'fs_csmask'. Add a comment to newfs/mkfs.c
to indicate that these fields must be calculated for compatibility
with older kernels.
Reviewed by: mckusick
a SIGINFO (normally via Ctrl-T), a line will be output indicating
the current phase number and progress information relevant to the
current phase.
Approved by: mckusick
