Commit Graph

105 Commits

Author SHA1 Message Date
Kirk McKusick
068beacf21 The goal of this change is to prevent accidental foot shooting by
folks running filesystems created on check-hash enabled kernels
(which I will call "new") on a non-check-hash enabled kernels (which
I will call "old). The idea here is to detect when a filesystem is
run on an old kernel and flag the filesystem so that when it gets
moved back to a new kernel, it will not start getting a slew of
check-hash errors.

Back when the UFS version 2 filesystem was created, it added a file
flag FS_INDEXDIRS that was to be set on any filesystem that kept
some sort of on-disk indexing for directories. The idea was precisely
to solve the issue we have today. Specifically that a newer kernel
that supported indexing would be able to tell that the filesystem
had been run on an older non-indexing kernel and that the indexes
should not be used until they had been rebuilt. Since we have never
implemented on-disk directory indicies, the FS_INDEXDIRS flag is
cleared every time any UFS version 2 filesystem ever created is
mounted for writing.

This commit repurposes the FS_INDEXDIRS flag as the FS_METACKHASH
flag. Thus, the FS_METACKHASH is definitively known to have always
been cleared. The FS_INDEXDIRS flag has been moved to a new block
of flags that will always be cleared starting with this commit
(until they get used to implement some future feature which needs
to detect that the filesystem was mounted on a kernel that predates
the new feature).

If a filesystem with check-hashes enabled is mounted on an old
kernel the FS_METACKHASH flag is cleared. When that filesystem is
mounted on a new kernel it will see that the FS_METACKHASH has been
cleared and clears all of the fs_metackhash flags. To get them
re-enabled the user must run fsck (in interactive mode without the
-y flag) which will ask for each supported check hash whether it
should be rebuilt and enabled. When fsck is run in its default preen
mode, it will just ignore the check hashes so they will remain
disabled.

The kernel has always disabled any check hash functions that it
does not support, so as more types of check hashes are added, we
will get a non-surprising result. Specifically if filesystems get
moved to kernels supporting fewer of the check hashes, those that
are not supported will be disabled. If the filesystem is moved back
to a kernel with more of the check-hashes available and fsck is run
interactively to rebuild them, then their checking will resume.
Otherwise just the smaller subset will be checked.

A side effect of this commit is that filesystems running with
cylinder-group check hashes will stop having them checked until
fsck is run to re-enable them (since none of them currently have
the FS_METACKHASH flag set). So, if you want check hashes enabled
on your filesystems after booting a kernel with these changes, you
need to run fsck to enable them. Any newly created filesystems will
have check hashes enabled. If in doubt as to whether you have check
hashes emabled, run dumpfs and look at the list of enabled flags
at the end of the superblock details.
2018-02-08 23:06:58 +00:00
Pedro F. Giffuni
8a16b7a18f General further adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 3-Clause license.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Special thanks to Wind River for providing access to "The Duke of
Highlander" tool: an older (2014) run over FreeBSD tree was useful as a
starting point.
2017-11-20 19:49:47 +00:00
Bryan Drewery
ea825d0274 DIRDEPS_BUILD: Update dependencies.
Sponsored by:	Dell EMC Isilon
2017-10-31 00:07:04 +00:00
Kirk McKusick
75e3597abb Continuing efforts to provide hardening of FFS, this change adds a
check hash to cylinder groups. If a check hash fails when a cylinder
group is read, no further allocations are attempted in that cylinder
group until it has been fixed by fsck. This avoids a class of
filesystem panics related to corrupted cylinder group maps. The
hash is done using crc32c.

Check hases are added only to UFS2 and not to UFS1 as UFS1 is primarily
used in embedded systems with small memories and low-powered processors
which need as light-weight a filesystem as possible.

Specifics of the changes:

sys/sys/buf.h:
    Add BX_FSPRIV to reserve a set of eight b_xflags that may be used
    by individual filesystems for their own purpose. Their specific
    definitions are found in the header files for each filesystem
    that uses them. Also add fields to struct buf as noted below.

sys/kern/vfs_bio.c:
    It is only necessary to compute a check hash for a cylinder
    group when it is actually read from disk. When calling bread,
    you do not know whether the buffer was found in the cache or
    read. So a new flag (GB_CKHASH) and a pointer to a function to
    perform the hash has been added to breadn_flags to say that the
    function should be called to calculate a hash if the data has
    been read. The check hash is placed in b_ckhash and the B_CKHASH
    flag is set to indicate that a read was done and a check hash
    calculated. Though a rather elaborate mechanism, it should
    also work for check hashing other metadata in the future. A
    kernel internal API change was to change breada into a static
    fucntion and add flags and a function pointer to a check-hash
    function.

sys/ufs/ffs/fs.h:
    Add flags for types of check hashes; stored in a new word in the
    superblock. Define corresponding BX_ flags for the different types
    of check hashes. Add a check hash word in the cylinder group.

sys/ufs/ffs/ffs_alloc.c:
    In ffs_getcg do the dance with breadn_flags to get a check hash and
    if one is provided, check it.

sys/ufs/ffs/ffs_vfsops.c:
    Copy across the BX_FFSTYPES flags in background writes.
    Update the check hash when writing out buffers that need them.

sys/ufs/ffs/ffs_snapshot.c:
    Recompute check hash when updating snapshot cylinder groups.

sys/libkern/crc32.c:
lib/libufs/Makefile:
lib/libufs/libufs.h:
lib/libufs/cgroup.c:
    Include libkern/crc32.c in libufs and use it to compute check
    hashes when updating cylinder groups.

Four utilities are affected:

sbin/newfs/mkfs.c:
    Add the check hashes when building the cylinder groups.

sbin/fsck_ffs/fsck.h:
sbin/fsck_ffs/fsutil.c:
    Verify and update check hashes when checking and writing cylinder groups.

sbin/fsck_ffs/pass5.c:
    Offer to add check hashes to existing filesystems.
    Precompute check hashes when rebuilding cylinder group
    (although this will be done when it is written in fsutil.c
    it is necessary to do it early before comparing with the old
    cylinder group)

sbin/dumpfs/dumpfs.c
    Print out the new check hash flag(s)

sbin/fsdb/Makefile:
    Needs to add libufs now used by pass5.c imported from fsck_ffs.

Reviewed by: kib
Tested by: Peter Holm (pho)
2017-09-22 12:45:15 +00:00
Warner Losh
fbbd9655e5 Renumber copyright clause 4
Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.

Submitted by:	Jan Schaumann <jschauma@stevens.edu>
Pull Request:	https://github.com/freebsd/freebsd/pull/96
2017-02-28 23:42:47 +00:00
Marcelo Araujo
2c953857dd Use MIN macro from sys/param.h.
MFC after:	2 weeks
2016-05-02 01:40:31 +00:00
Glen Barber
406d87b1c3 Explicitly add more files to the 'runtime' package.
Sponsored by:	The FreeBSD Foundation
2016-02-09 20:19:31 +00:00
Edward Tomasz Napierala
2f221c70f9 The dumpfs(8) utility doesn't actually use disktab(5); remove it
from "SEE ALSO".

MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
2015-09-03 11:40:47 +00:00
Edward Tomasz Napierala
b8c19fd719 It's 2015, and some people are still trying to use fdisk and then
go asking what debug flags to set for GEOM to make it work.  Advice
them to use gpart(8) instead.

Something similar should probably done with disklabel,
but I need to rewrite the disklabel examples first.

Reviewed by:	wblock@
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3315
2015-09-02 14:08:43 +00:00
Simon J. Gerraty
ccfb965433 Add META_MODE support.
Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision:       D2796
Reviewed by: brooks imp
2015-06-13 19:20:56 +00:00
Simon J. Gerraty
44d314f704 dirdeps.mk now sets DEP_RELDIR 2015-06-08 23:35:17 +00:00
Simon J. Gerraty
98e0ffaefb Merge sync of head 2015-05-27 01:19:58 +00:00
Andrey V. Elsukov
24211a0afc Print leading zeroes of UFS2 fs_id like we do for UFS1.
PR:		156908
MFC after:	1 week
2015-05-26 18:27:38 +00:00
Baptiste Daroussin
13eb765f2d Convert sbin/ to LIBADD
Reduce overlinking
2014-11-25 11:23:12 +00:00
Simon J. Gerraty
fae50821ae Updated dependencies 2014-05-16 14:09:51 +00:00
Simon J. Gerraty
76b28ad6ab Updated dependencies 2014-05-10 05:16:28 +00:00
Simon J. Gerraty
d1d0158641 Merge from head 2013-09-05 20:18:59 +00:00
Kirk McKusick
9d384d4d3d When running the -m option to generate a newfs(8) command suitable for
recreating the filesystem, check for and output the -i, -k, and -l
options if appropriate.

Note the remaining deficiencies of the -m option in the dumpfs(8)
manual page. Specifically that newfs(8) options -E, -R, -S, and -T
options are not handled and that -p is not useful so is omitted.
Also document that newfs(8) options -n and -r are neither checked
for nor output but should be. The -r flag is needed if the filesystem
uses gjournal(8).

PR:           bin/163992
Reported by:  Dieter <freebsd@sopwith.solgatos.com>
Submitted by: Andy Kosela <akosela@andykosela.com>
MFC after:    1 week
2013-05-16 20:07:08 +00:00
Kirk McKusick
3e8b845956 Clean up trailing whitespace.
Submitted by: Andy Kosela
MFC after:    1 week
2013-05-16 19:39:11 +00:00
Simon J. Gerraty
69e6d7b75e sync from head 2013-04-12 20:48:55 +00:00
Kirk McKusick
baa12a84a7 The purpose of this change to the FFS layout policy is to reduce the
running time for a full fsck. It also reduces the random access time
for large files and speeds the traversal time for directory tree walks.

The key idea is to reserve a small area in each cylinder group
immediately following the inode blocks for the use of metadata,
specifically indirect blocks and directory contents. The new policy
is to preferentially place metadata in the metadata area and
everything else in the blocks that follow the metadata area.

The size of this area can be set when creating a filesystem using
newfs(8) or changed in an existing filesystem using tunefs(8).
Both utilities use the `-k held-for-metadata-blocks' option to
specify the amount of space to be held for metadata blocks in each
cylinder group. By default, newfs(8) sets this area to half of
minfree (typically 4% of the data area).

This work was inspired by a paper presented at Usenix's FAST '13:
www.usenix.org/conference/fast13/ffsck-fast-file-system-checker

Details of this implementation appears in the April 2013 of ;login:
www.usenix.org/publications/login/april-2013-volume-38-number-2.
A copy of the April 2013 ;login: paper can also be downloaded
from: www.mckusick.com/publications/faster_fsck.pdf.

Reviewed by: kib
Tested by:   Peter Holm
MFC after:   4 weeks
2013-03-22 21:45:28 +00:00
Simon J. Gerraty
7cf3a1c6b2 Updated dependencies 2013-03-11 17:21:52 +00:00
Simon J. Gerraty
f5f7c05209 Updated dependencies 2013-02-16 01:23:54 +00:00
Simon J. Gerraty
23090366f7 Sync from head 2012-11-04 02:52:03 +00:00
Edward Tomasz Napierala
549f62fa42 Fix problem with geom_label(4) not recognizing UFS labels on filesystems
extended using growfs(8).  The problem here is that geom_label checks if
the filesystem size recorded in UFS superblock is equal to the provider
(i.e. device) size.  This check cannot be removed due to backward
compatibility.  On the other hand, in most cases growfs(8) cannot set
fs_size in the superblock to match the provider size, because, differently
from newfs(8), it cannot recompute cylinder group sizes.

To fix this problem, add another superblock field, fs_providersize, used
only for this purpose.  The geom_label(4) will attach if either fs_size
(filesystem created with newfs(8)) or fs_providersize (filesystem expanded
using growfs(8)) matches the device size.

PR:		kern/165962
Reviewed by:	mckusick
Sponsored by:	FreeBSD Foundation
2012-10-30 21:32:10 +00:00
Eitan Adler
73ea379e32 Add support for gjournal to dumpfs
PR:		165821
Submitted by:	Andreas Longwitz <longwitz@incore.de>
Approved by:	cperciva
MFC after:	3 days
2012-10-22 03:00:20 +00:00
Marcel Moolenaar
7750ad47a9 Sync FreeBSD's bmake branch with Juniper's internal bmake branch.
Requested by: Simon Gerraty <sjg@juniper.net>
2012-08-22 19:25:57 +00:00
Christian Brueffer
4ff79fdf3f Add missing -l flag to usage().
PR:		163629
Submitted by:	olgeni
MFC after:	1 week
2011-12-26 16:47:45 +00:00
Ed Schouten
2162cff196 Staticify dumpfs.
This tool only consists of a single C file, so we can simply mark
everything except main() static. This seems to shave off about 8% of the
binary size.
2011-12-13 09:01:44 +00:00
Ed Schouten
1efe3c6b58 Add missing static keywords for global variables to tools in sbin/.
These tools declare global variables without using the static keyword,
even though their use is limited to a single C-file, or without placing
an extern declaration of them in the proper header file.
2011-11-04 13:36:02 +00:00
Xin LI
ef84bb4cd7 Use _PATH_DEV and make the format more consistent with GEOM_LABEL.
Submitted by:	ivoras
2011-07-14 17:43:23 +00:00
Xin LI
e293d08775 Add a -l option to show file system's corresponding /dev/ufsid path.
This is useful for scripts that converts existing system's fstab to
use their /dev/ufsid devices.

MFC after:	2 weeks
2011-07-14 08:06:03 +00:00
Gavin Atkinson
275e1cf79b We now have multiple filesystems (UFS, ZFS, ...), so for tools that only
operate on one type of filesystem, mention this.
While here, capitalise the use of "UFS" in growfs.8 to match other uses of
the term in other man pages.

MFC after:	1 week
2011-05-08 12:34:31 +00:00
Juli Mallett
9775d5ebdf Add support for displaying newfs flags for SU+J and TRIM. 2011-03-23 08:33:12 +00:00
Konstantin Belousov
a738d4cf20 Add support for FS_TRIM to user-mode UFS utilities.
Reviewed by:	mckusick, pjd, pho
Tested by:	pho
MFC after:	1 month
2010-12-29 12:31:18 +00:00
Jeff Roberson
113db2dddb - Merge soft-updates journaling from projects/suj/head into head. This
brings in support for an optional intent log which eliminates the need
   for background fsck on unclean shutdown.

Sponsored by:   iXsystems, Yahoo!, and Juniper.
With help from: McKusick and Peter Holm
2010-04-24 07:05:35 +00:00
Gavin Atkinson
8f70111670 Print leading zeros in the UFS2 FSID.
PR:		bin/142155
Submitted by:	Efstratios Karatzas  gpf.kira gmail.com
Approved by:	ed (mentor)
MFC after:	2 weeks
2010-01-06 14:01:28 +00:00
Edward Tomasz Napierala
9340fc72e6 Implement NFSv4 ACL support for UFS.
Reviewed by:	rwatson
2009-12-21 19:39:10 +00:00
Ruslan Ermilov
106d839190 Switch the default WARNS level for sbin/ to 6.
Submitted by:	Ulrich Spörlein
2009-10-19 16:00:24 +00:00
Ruslan Ermilov
f807cca1a3 Properly re-create "-s size" argument to newfs(8). 2009-10-19 14:04:19 +00:00
Robert Watson
6458691cc4 Print disk offets as %jd rather than %lld; I fixed one before committing
but missed the other, which breaks 64-bit builds.

Reported by:	bf <bf2006a at yahoo dot com>
MFC after:	1 week
2009-01-28 12:08:19 +00:00
Tom Rhodes
4b797ca8d7 Bump doc date for recent change. 2009-01-28 11:31:09 +00:00
Robert Watson
fd15d586ee Add a new flag to dumpfs(8), -f, which causes dumpfs to list all free
fragments in the file system by fragment (block) number.  This new
mode does the necessary arithmetic to generate absolute fragment
numbers rather than than the cg-relative numbers printed in the default
mode.

If -f is passed once, contiguous fragment ranges are collapsed into
an X-Y format as free block lists are currently printed in regular
dumpfs output, but if specified twice, all block numbers are printed
individually, allowing both compact and more script-friendly
representation.

This proves quite handy when attempting to recover deleted data, as it
allows exclusion of non-deleted data from blocks searched.
MFC after:	1 week
Discussed with:	jeff, Richard Clayton <richard dot clayton at cl.cam.ac.uk>
Sponsored by:	Google, Inc.
2009-01-28 09:33:00 +00:00
Pav Lucistnik
a8dec2ad79 Replace incomprehensive description of -m by much clearer text from OpenBSD,
with the exception of one word.

PR:		docs/112465
Submitted by:	naddy
Obtained from:	OpenBSD
MFC after:	1 week
2007-05-12 22:35:22 +00:00
Pawel Jakub Dawidek
5a9dc069e0 Teach about new fields (cg_unrefs and fs_unrefs) and new FS_GJOURNAL flag.
Sponsored by:	home.pl
2006-10-31 22:02:24 +00:00
Ruslan Ermilov
8d646af581 Sync program's usage() with manpage's SYNOPSIS. 2005-02-10 09:19:34 +00:00
Ruslan Ermilov
9806e23132 Mechanically kill hard sentence breaks. 2004-07-02 21:45:06 +00:00
Mark Murray
4c723140a4 Remove advertising clause from University of California Regent's license,
per letter dated July 22, 1999.

Approved by: core, imp
2004-04-09 19:58:40 +00:00
Don Lewis
0482f576f1 Reinstate 1.40 -- swap avgfilesize and avgfpdir column order.
MFC after:	3 days
2003-12-07 05:27:27 +00:00
Don Lewis
408145888e I forgot about the code freeze, so back this out. 2003-12-01 00:33:57 +00:00