Commit Graph

123 Commits

Author SHA1 Message Date
Kirk McKusick
5cc52631b3 Rewrite the disk I/O management system in fsck_ffs(8). Other than
making fsck_ffs(8) run faster, there should be no functional change.

The original fsck_ffs(8) had its own disk I/O management system.
When gjournal(8) was added to FreeBSD 7, code was added to fsck_ffs(8)
to do the necessary gjournal rollback. Rather than use the existing
fsck_ffs(8) disk I/O system, it wrote its own from scratch. Similarly
when journalled soft updates were added in FreeBSD 9, code was added
to fsck_ffs(8) to do the necessary journal rollback. And once again,
rather than using either of the existing fsck_ffs(8) disk I/O
systems, it wrote its own from scratch. Lastly the fsdb(8) utility
uses the fsck_ffs(8) disk I/O management system. In preparation for
making the changes necessary to enable snapshots to be taken when
using journalled soft updates, it was necessary to have a single
disk I/O system used by all the various subsystems in fsck_ffs(8).

This commit merges the functionality required by all the different
subsystems into a single disk I/O system that supports all of their
needs. In so doing it picks up optimizations from each of them
with the results that each of the subsystems does fewer reads and
writes than it did with its own customized I/O system. It also
greatly simplifies making changes to fsck_ffs(8) since everything
goes through a single place. For example the ginode() function
fetches an inode from the disk. When inode check hashes were added,
they previously had to be checked in the code implementing inode
fetch in each of the three different disk I/O systems. Now they
need only be checked in ginode().

Tested by:    Peter Holm
Sponsored by: Netflix
2021-01-07 15:03:15 -08:00
Gordon Bergling
a401a76ceb fsdb(8): Fix an issue reported by mandoc
- whitespace at end of input line

MFC after:	1 week
2020-10-03 18:44:13 +00:00
Kirk McKusick
68d7185b64 The fsdb(8) utility uses the fsck_ffs(8) disk I/O interfaces, so
switch from using libufs's bread() to using fsck_ffs's getdatablk()
when importing tools/diag/prtblnos's prtblknos().

Sponsored by: Netflix
2020-09-19 20:06:12 +00:00
Kyle Evans
c3e9752ea1 fsck_ffs/fsdb: fix -fno-common build
This one is also a small list:

- 3x duplicate definition (ufs2_zino, returntosingle, nflag)
- 5x 'needs extern', 3/5 of which are referenced in fsdb

-fno-common will become the default in GCC10/LLVM11.

MFC after:	1 week
2020-03-29 20:03:46 +00:00
Kirk McKusick
ac4b20a0a7 After a crash, a file that extends into indirect blocks may end up
shorter than its size resulting in a hole as its final block (which
is a violation of the invarients of the UFS filesystem).

Soft updates will always ensure that the file size is correct when
writing inodes to disk for files that contain only direct block
pointers. However soft updates does not roll back sizes for files
with indirect blocks that it has set to unallocated because their
contents have not yet been written to disk. Hence, the file can
appear to have a hole at its end because the block pointer has been
rolled back to zero when its inode was written to disk. Thus,
fsck_ffs calculates the last allocated block in the file. For files
that extend into indirect blocks, fsck_ffs checks for a size past
the last allocated block of the file and if that is found, shortens
the file to reference the last allocated block thus avoiding having
it reference a hole at its end.

Submitted by: Chuck Silvers <chs@netflix.com>
Tested by:    Chuck Silvers <chs@netflix.com>
MFC after:    1 week
Sponsored by: Netflix
2019-02-25 21:58:19 +00:00
Kirk McKusick
2c288c95d9 In preparation for adding inode check-hashes, change the fsck_ffs
inodirty() function to have a pointer to the inode being dirtied.
No functional change (as for now the parameter is ununsed).

Sponsored by: Netflix
2018-10-31 05:17:53 +00:00
Kirk McKusick
2a36aee276 When using the fsdb `blocks' command, replace the long and ugly list of
blocks with the much more concise and readable block list shown by the
prtblknos() function imported from tools/diag/prtblknos.
2018-04-08 07:06:12 +00:00
Ed Maste
d8ba45e213 Revert r313780 (UFS_ prefix) 2018-03-17 12:59:55 +00:00
Ed Maste
1e2b9afca9 Prefix UFS symbols with UFS_ to reduce namespace pollution
Followup to r313780.  Also prefix ext2's and nandfs's versions with
EXT2_ and NANDFS_.

Reported by:	kib
Reviewed by:	kib, mckusick
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D9623
2018-03-17 01:48:27 +00:00
Dimitry Andric
2d3c3a5038 Fix buildworld after r328075, by also renaming cgget to cglookup in
fsdb.

Reported by:	ohartmann@walstatt.org,david@catwhisker.org
Pointy hat to:	mckusick
2018-01-17 13:19:37 +00:00
Xin LI
5a8ad265c9 The fix in r327273 turns a memory leak into freeing wild pointer.
Fix this by freeing only the initialized pointer.
2017-12-29 06:22:05 +00:00
Warner Losh
0bebba31d3 Plug memory leak by freeing wantedblk{32,64}.
CID: 273655, 273656
2017-12-28 05:34:08 +00:00
Pedro F. Giffuni
1de7b4b805 various: general adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.
2017-11-27 15:37:16 +00:00
Bryan Drewery
ea825d0274 DIRDEPS_BUILD: Update dependencies.
Sponsored by:	Dell EMC Isilon
2017-10-31 00:07:04 +00:00
Kirk McKusick
75e3597abb Continuing efforts to provide hardening of FFS, this change adds a
check hash to cylinder groups. If a check hash fails when a cylinder
group is read, no further allocations are attempted in that cylinder
group until it has been fixed by fsck. This avoids a class of
filesystem panics related to corrupted cylinder group maps. The
hash is done using crc32c.

Check hases are added only to UFS2 and not to UFS1 as UFS1 is primarily
used in embedded systems with small memories and low-powered processors
which need as light-weight a filesystem as possible.

Specifics of the changes:

sys/sys/buf.h:
    Add BX_FSPRIV to reserve a set of eight b_xflags that may be used
    by individual filesystems for their own purpose. Their specific
    definitions are found in the header files for each filesystem
    that uses them. Also add fields to struct buf as noted below.

sys/kern/vfs_bio.c:
    It is only necessary to compute a check hash for a cylinder
    group when it is actually read from disk. When calling bread,
    you do not know whether the buffer was found in the cache or
    read. So a new flag (GB_CKHASH) and a pointer to a function to
    perform the hash has been added to breadn_flags to say that the
    function should be called to calculate a hash if the data has
    been read. The check hash is placed in b_ckhash and the B_CKHASH
    flag is set to indicate that a read was done and a check hash
    calculated. Though a rather elaborate mechanism, it should
    also work for check hashing other metadata in the future. A
    kernel internal API change was to change breada into a static
    fucntion and add flags and a function pointer to a check-hash
    function.

sys/ufs/ffs/fs.h:
    Add flags for types of check hashes; stored in a new word in the
    superblock. Define corresponding BX_ flags for the different types
    of check hashes. Add a check hash word in the cylinder group.

sys/ufs/ffs/ffs_alloc.c:
    In ffs_getcg do the dance with breadn_flags to get a check hash and
    if one is provided, check it.

sys/ufs/ffs/ffs_vfsops.c:
    Copy across the BX_FFSTYPES flags in background writes.
    Update the check hash when writing out buffers that need them.

sys/ufs/ffs/ffs_snapshot.c:
    Recompute check hash when updating snapshot cylinder groups.

sys/libkern/crc32.c:
lib/libufs/Makefile:
lib/libufs/libufs.h:
lib/libufs/cgroup.c:
    Include libkern/crc32.c in libufs and use it to compute check
    hashes when updating cylinder groups.

Four utilities are affected:

sbin/newfs/mkfs.c:
    Add the check hashes when building the cylinder groups.

sbin/fsck_ffs/fsck.h:
sbin/fsck_ffs/fsutil.c:
    Verify and update check hashes when checking and writing cylinder groups.

sbin/fsck_ffs/pass5.c:
    Offer to add check hashes to existing filesystems.
    Precompute check hashes when rebuilding cylinder group
    (although this will be done when it is written in fsutil.c
    it is necessary to do it early before comparing with the old
    cylinder group)

sbin/dumpfs/dumpfs.c
    Print out the new check hash flag(s)

sbin/fsdb/Makefile:
    Needs to add libufs now used by pass5.c imported from fsck_ffs.

Reviewed by: kib
Tested by: Peter Holm (pho)
2017-09-22 12:45:15 +00:00
Enji Cooper
22289a8c3d sbin: normalize paths using SRCTOP-relative paths or :H when possible
This simplifies make logic/output

MFC after:	1 month
Sponsored by:	Dell EMC Isilon
2017-03-04 11:33:01 +00:00
Ed Maste
1dc349ab95 prefix UFS symbols with UFS_ to reduce namespace pollution
Specifically:
  ROOTINO -> UFS_ROOTINO
  WINO -> UFS_WINO
  NXADDR -> UFS_NXADDR
  NDADDR -> UFS_NDADDR
  NIADDR -> UFS_NIADDR
  MAXSYMLINKLEN_UFS[12] -> UFS[12]_MAXSYMLINKLEN (for consistency)

Also prefix ext2's and nandfs's NDADDR and NIADDR with EXT2_ and NANDFS_

Reviewed by:	kib, mckusick
Obtained from:	NetBSD
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D9536
2017-02-15 19:50:26 +00:00
Sevan Janiyan
369cef877e Document which version of BSD first showed up in and add the version info
for NetBSD & FreeBSD.

PR:		212477
Approved by:	bcr (mentor)
MFC after:	4 days
Differential Revision:	https://reviews.freebsd.org/D8105
2016-10-02 23:44:52 +00:00
Marcelo Araujo
02c8c1182b Use nitems() from sys/param.h.
Sponsored by:	gandi.net (BSD Day Taiwan)
2016-07-30 06:19:34 +00:00
Marcelo Araujo
7fd35136d9 Use NULL instead of 0 for pointers.
malloc will return NULL if it cannot allocate memory.

MFC after:	2 weeks.
2016-04-18 07:47:26 +00:00
Glen Barber
406d87b1c3 Explicitly add more files to the 'runtime' package.
Sponsored by:	The FreeBSD Foundation
2016-02-09 20:19:31 +00:00
Simon J. Gerraty
ccfb965433 Add META_MODE support.
Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision:       D2796
Reviewed by: brooks imp
2015-06-13 19:20:56 +00:00
Simon J. Gerraty
44d314f704 dirdeps.mk now sets DEP_RELDIR 2015-06-08 23:35:17 +00:00
Simon J. Gerraty
98e0ffaefb Merge sync of head 2015-05-27 01:19:58 +00:00
Baptiste Daroussin
13eb765f2d Convert sbin/ to LIBADD
Reduce overlinking
2014-11-25 11:23:12 +00:00
Simon J. Gerraty
ee7b0571c2 Merge head from 7/28 2014-08-19 06:50:54 +00:00
Brooks Davis
80189b3b09 Replace all uses of libncurses and libtermcap with their wide character
variants.  This allows usable file system images (i.e. those with both a
shell and an editor) to be created with only one copy of the curses library.

Exp-run:	antoine
PR:		189842
Discussed with:	bapt
Sponsored by:	DARPA, AFRL
2014-07-17 18:24:34 +00:00
Simon J. Gerraty
fae50821ae Updated dependencies 2014-05-16 14:09:51 +00:00
Simon J. Gerraty
76b28ad6ab Updated dependencies 2014-05-10 05:16:28 +00:00
Simon J. Gerraty
3b8f084595 Merge head 2014-04-28 07:50:45 +00:00
Scott Long
9a2fcb2e6b Add globs.c to the build now that it's a separate file.
Reviewed by:	max
Obtained from:	Netflix
MFC after:	3 days
2013-12-30 01:17:05 +00:00
Simon J. Gerraty
69e6d7b75e sync from head 2013-04-12 20:48:55 +00:00
Kirk McKusick
81fbded23f Revert 248634 and 248643 (e.g., restoring 248625 and 248639).
Build verified by: Glen Barber (gjb@)
2013-03-23 20:00:02 +00:00
Glen Barber
e1c7cfb200 Revert r248639 to fix build failure on head/ 2013-03-23 08:57:14 +00:00
Kirk McKusick
25bcd53216 Fix the build after addition of cylinder group cacheing (r248625)
Reported by:   Glen Barber (gjb@)
Pointy hat to: Kirk McKusick (mckusick@)
2013-03-23 07:57:30 +00:00
Simon J. Gerraty
7cf3a1c6b2 Updated dependencies 2013-03-11 17:21:52 +00:00
Sergey Kandaurov
3dfcb0da4b Catch up with internal API changes for initbarea() and getdatablk()
of fsck_ffs introduced with r247212.

Submitted by:	David Wolfskill <david@catwhisker.org>
2013-02-24 19:32:43 +00:00
Simon J. Gerraty
f5f7c05209 Updated dependencies 2013-02-16 01:23:54 +00:00
Sergey Kandaurov
fa358104b1 Remove write only assignments and thus fix the build after struct bufarea
TAILQ conversion (r246812).
2013-02-15 09:10:01 +00:00
Simon J. Gerraty
23090366f7 Sync from head 2012-11-04 02:52:03 +00:00
Matthew D Fleming
e25a029eb2 Fix sbin/ build with a 64-bit ino_t.
Original code by:	Gleb Kurtsou
2012-09-27 23:31:06 +00:00
Marcel Moolenaar
7750ad47a9 Sync FreeBSD's bmake branch with Juniper's internal bmake branch.
Requested by: Simon Gerraty <sjg@juniper.net>
2012-08-22 19:25:57 +00:00
Dimitry Andric
24c8f29cf4 Partially undo r228693, by removing NO_WFORMAT.clang in fsdb's Makefile,
and fixing the format string in sbin/fsdb/fsdbutil.c instead.

Note the remark "Work around a problem with format string warnings and
ntohs macros" was actually incorrect.  The DIP(dp, di_nlink) macro
invocation actually returned an int, due to its ternary expression, even
though the di_nlink members of struct ufs1_dinode and struct ufs2_dinode
are both defined as int16_t.

MFC after:	2 weeks
2012-03-09 21:02:39 +00:00
Dimitry Andric
07b202a847 Define several extra macros in bsd.sys.mk and sys/conf/kern.pre.mk, to
get rid of testing explicitly for clang (using ${CC:T:Mclang}) in
individual Makefiles.

Instead, use the following extra macros, for use with clang:
- NO_WERROR.clang       (disables -Werror)
- NO_WCAST_ALIGN.clang  (disables -Wcast-align)
- NO_WFORMAT.clang	(disables -Wformat and friends)
- CLANG_NO_IAS		(disables integrated assembler)
- CLANG_OPT_SMALL	(adds flags for extra small size optimizations)

As a side effect, this enables setting CC/CXX/CPP in src.conf instead of
make.conf!  For clang, use the following:

CC=clang
CXX=clang++
CPP=clang-cpp

MFC after:	2 weeks
2012-02-28 18:30:18 +00:00
Don Lewis
ec1a0f73d0 Improve sparse file handling when printing the block list for an inode by
not bailing out early when a hole is encountered in the direct block list.
Print NULL block pointers in the direct block list.  Simplify the
code that prints the fragment count.

Match the style of the existing code.

Reviewed by:	mckusick
MFC after:	1 week
2012-02-06 21:50:11 +00:00
Dimitry Andric
cf2cbe3458 Revert r228603, and add the workaround to sbin/fsdb/Makefile instead.
MFC after:	1 week
2011-12-18 17:42:26 +00:00
Dimitry Andric
9733a9e463 In sbin/fsdb/fsdbutil.c, work around a clang false positive with printf
format warnings and conditional operators.  (See LLVM PR 11313 for more
information.)

MFC after:	1 week
2011-12-16 23:09:31 +00:00
Ed Schouten
9f365aa1d6 Get rid of major/minor number distinction.
As of FreeBSD 6, devices can only be opened through devfs. These device
nodes don't have major and minor numbers anymore. The st_rdev field in
struct stat is simply based a copy of st_ino.

Simply display device numbers as hexadecimal, using "%#jx". This is
allowed by POSIX, since it explicitly states things like the following
(example taken from ls(1)):

	"If the file is a character special or block special file, the
	size of the file may be replaced with implementation-defined
	information associated with the device in question."

This makes the output of these commands more compact. For example, ls(1)
now uses approximately four columns less. While there, simplify the
column length calculation from ls(1) by calling snprintf() with a NULL
buffer.

Don't be afraid; if needed one can still obtain individual major/minor
numbers using stat(1).
2011-09-28 18:53:36 +00:00
Kirk McKusick
7649cb0043 The dump, fsck_ffs, fsdb, fsirand, newfs, makefs, and quot utilities
include sys/time.h instead of time.h. This include is incorrect as
per the manpages for the APIs and the POSIX definitions. This commit
replaces sys/time.h where necessary with time.h.

The commit also includes some minor style(9) header fixup in newfs.

This commit is part of a larger effort by Garrett Cooper started in
//depot/user/gcooper/posix-conformance-work/ -- to make FreeBSD more
POSIX compliant.

Submitted by:  Garrett Cooper   yanegomi at gmail dot com
2011-01-24 06:17:05 +00:00
Jeff Roberson
113db2dddb - Merge soft-updates journaling from projects/suj/head into head. This
brings in support for an optional intent log which eliminates the need
   for background fsck on unclean shutdown.

Sponsored by:   iXsystems, Yahoo!, and Juniper.
With help from: McKusick and Peter Holm
2010-04-24 07:05:35 +00:00