Commit Graph

60 Commits

Author SHA1 Message Date
Xin LI
1b83e8a3f8 Constify string pointers.
Verified with:	sha256(1)
MFC after:	2 weeks
2013-05-16 21:04:56 +00:00
Kirk McKusick
244dccb7fd Fix error check.
Submitted by: Andrey Chernov (ache@)
MFC after: 3 days
2013-04-23 06:37:50 +00:00
Kirk McKusick
baa12a84a7 The purpose of this change to the FFS layout policy is to reduce the
running time for a full fsck. It also reduces the random access time
for large files and speeds the traversal time for directory tree walks.

The key idea is to reserve a small area in each cylinder group
immediately following the inode blocks for the use of metadata,
specifically indirect blocks and directory contents. The new policy
is to preferentially place metadata in the metadata area and
everything else in the blocks that follow the metadata area.

The size of this area can be set when creating a filesystem using
newfs(8) or changed in an existing filesystem using tunefs(8).
Both utilities use the `-k held-for-metadata-blocks' option to
specify the amount of space to be held for metadata blocks in each
cylinder group. By default, newfs(8) sets this area to half of
minfree (typically 4% of the data area).

This work was inspired by a paper presented at Usenix's FAST '13:
www.usenix.org/conference/fast13/ffsck-fast-file-system-checker

Details of this implementation appears in the April 2013 of ;login:
www.usenix.org/publications/login/april-2013-volume-38-number-2.
A copy of the April 2013 ;login: paper can also be downloaded
from: www.mckusick.com/publications/faster_fsck.pdf.

Reviewed by: kib
Tested by:   Peter Holm
MFC after:   4 weeks
2013-03-22 21:45:28 +00:00
Peter Holm
14951d4234 The .journal file needs to reside on the ROOTINO which must not extend
beyond direct blocks. A typo caused this check to fail.
2013-02-27 18:12:04 +00:00
Edward Tomasz Napierala
05d43d9882 Declare functions as static and move global variables to the top;
no functional changes.
2012-10-28 19:38:42 +00:00
Matthew D Fleming
e25a029eb2 Fix sbin/ build with a 64-bit ino_t.
Original code by:	Gleb Kurtsou
2012-09-27 23:31:06 +00:00
Eitan Adler
14f6494f61 Fix warning when compiling with gcc46:
error: variable 'Sflag' set but not used

Approved by:	dim
MFC after:	3 days
2012-01-10 02:58:52 +00:00
Ed Schouten
1efe3c6b58 Add missing static keywords for global variables to tools in sbin/.
These tools declare global variables without using the static keyword,
even though their use is limited to a single C-file, or without placing
an extern declaration of them in the proper header file.
2011-11-04 13:36:02 +00:00
Kirk McKusick
8ee53ea3a1 After creating a filesystem using newfs -j the time stamps are all
zero and thus report as having been made in January 1970. Apart
from looking a bit silly, it also triggers alarms from scripts
that detect weird time stamps. This update sets all 4 (or 3, in
the case of UFS1) time stamps to the current time when enabling
journaling during newfs or later when enabling it with tunefs.

Reported by: Hans Ottevanger <hans@beastielabs.net>
MFC after:   1 week
2011-10-11 19:03:57 +00:00
Konstantin Belousov
e605011a00 When creating a directory entry for the journal, always read at least
the fragment, and write the full block. Reading less might not work
due to device sector size bigger then size of direntries in the
last directory fragment.

Reported by:	bz
In collaboration with:	pho
Reviewed by:	jeff
Tested by:	bz, pho
2011-02-12 13:12:45 +00:00
Konstantin Belousov
a738d4cf20 Add support for FS_TRIM to user-mode UFS utilities.
Reviewed by:	mckusick, pjd, pho
Tested by:	pho
MFC after:	1 month
2010-12-29 12:31:18 +00:00
Jeff Roberson
5305359514 - Round up the journal size to the block size so we don't confuse fsck.
Reported by:	Mikolaj Golub <to.my.trociny@gmail.com>

 - Only require 256k of blocks per-cg when trying to allocate contiguous
   journal blocks.  The storage may not actually be contiguous but is at
   least within one cg.
 - When disabling SUJ leave SU enabled and report this to the user.  It
   is expected that users will upgrade SU filesystems to SUJ and want
   a similar downgrade path.
2010-05-18 01:45:28 +00:00
Edwin Groothuis
727c128897 Improve usage of tunefs:
Document -j switch in usage() to reflect recent SUJ work.

Submitted by:   Alastair Hogge
MFC after:      1 week
2010-05-01 09:05:06 +00:00
Jeff Roberson
a6e09ef1c2 - Use the path to the filesystem mountpoint to look up the statfs
structure so that we correctly reload.  Note that tunefs doesn't
   properly detect the need to reload if the disk device is specified
   for a read-only mounted filesystem.
 - Lessen the contiguity requirement for the journal so that it is more
   likely to succeed.
2010-04-30 04:21:22 +00:00
Jeff Roberson
113db2dddb - Merge soft-updates journaling from projects/suj/head into head. This
brings in support for an optional intent log which eliminates the need
   for background fsck on unclean shutdown.

Sponsored by:   iXsystems, Yahoo!, and Juniper.
With help from: McKusick and Peter Holm
2010-04-24 07:05:35 +00:00
Kirk McKusick
a6cc0cf692 Quiet spurious warnings. 2010-02-11 06:33:35 +00:00
Edward Tomasz Napierala
9340fc72e6 Implement NFSv4 ACL support for UFS.
Reviewed by:	rwatson
2009-12-21 19:39:10 +00:00
Pawel Jakub Dawidek
868c68ed1d Add -J flag to both newfs(8) and tunefs(8) which allows to enable gjournal
support.
I left -j flag for UFS journal implementation which we may gain at some
point.

Sponsored by:	home.pl
2006-10-31 21:52:28 +00:00
Mark Murray
4c723140a4 Remove advertising clause from University of California Regent's license,
per letter dated July 22, 1999.

Approved by: core, imp
2004-04-09 19:58:40 +00:00
Bruce Evans
7644442481 Fixed some style bugs in the residue of rev.1.14 (mainly initialization in
declarations, uncuddled elses and excessive braces).
2004-03-26 16:11:13 +00:00
Bruce Evans
29f9611d67 Fixed some style bugs in or related to rev.1.13 (mainly misindentation of
the getopt() case statement).
2004-03-26 08:39:36 +00:00
David E. O'Brien
c69284ca08 Use __FBSDID() to quiet GCC 3.3 warnings. 2003-05-03 18:41:59 +00:00
Ruslan Ermilov
1f6a46318d Sort options. 2003-02-23 01:50:07 +00:00
Gordon Tetlow
c715b047bc Bring in support for volume labels to the filesystem utilities.
Reviewed by:	mckusick
2003-02-01 04:17:10 +00:00
Juli Mallett
907db4dd23 Fix problems with how libufs was used, with regard to mounted/active fs's,
in the new world order of libufs, where we also do statfs, and add a missing
close.
2003-01-28 02:42:01 +00:00
Juli Mallett
b1f0fda09f Make tunefs use libufs, it seems to do well enough for printing / setting
things.
2003-01-20 21:15:02 +00:00
Juli Mallett
e0328ede47 Consistentify output whitespace. 2003-01-18 06:29:15 +00:00
Kirk McKusick
ada981b228 Create a new 32-bit fs_flags word in the superblock. Add code to move
the old 8-bit fs_old_flags to the new location the first time that the
filesystem is mounted by a new kernel. One of the unused flags in
fs_old_flags is used to indicate that the flags have been moved.
Leave the fs_old_flags word intact so that it will work properly if
used on an old kernel.

Change the fs_sblockloc superblock location field to be in units
of bytes instead of in units of filesystem fragments. The old units
did not work properly when the fragment size exceeeded the superblock
size (8192). Update old fs_sblockloc values at the same time that
the flags are moved.

Suggested by:	BOUWSMA Barry <freebsd-misuser@netscum.dyndns.dk>
Sponsored by:   DARPA & NAI Labs.
2002-11-27 02:18:58 +00:00
Robert Watson
273500c258 s/clear/cleared/ for consistency (sigh)
Reported by:	dd
2002-10-16 05:03:40 +00:00
Robert Watson
c2cd97a3d0 Spell 'set' as 'cleared' where appropriate. 2002-10-15 21:23:22 +00:00
Robert Watson
81dc101cf6 Teach tunefs to print the ACL and multilabel flag information when
inspecting a superblock.

Obtained from:	TrustedBSD Project
Sponsored by:	DARPA, Network Associates Laboratories
2002-10-15 18:14:05 +00:00
Robert Watson
a2325efeb1 Correct some of the style problems in this file:
I introduced a style problem when I sorted 'a' before 'A'; our
preferred order sorts 'A' first.  Correct.

Use .Cm instead of .Ar.

Submitted by:	bde
2002-10-15 15:30:55 +00:00
Robert Watson
289e09ee73 Introduce -a [enable|disable] and -l [enable|disable] flags to the tunefs
command, permitting it to set FS_ACLS and FS_MULTILABEL administrative
flags on UFS file systems.

Obtained from:	TrustedBSD Project
Sponsored by:	DARPA, Network Associates Laboratories
2002-10-14 19:52:12 +00:00
Bruce Evans
23d8e0317a Removed vestiges of the -a and -d options.
Fixed other bugs in the usage message so that it matches the man page.
2002-09-06 13:16:24 +00:00
Poul-Henning Kamp
a9098c8910 Remove the -a maxcontig option, the kernel doesn't inspect fs_maxcontig
anymore.

Sponsored by:	DARPA & NAI Labs.
2002-09-06 07:59:10 +00:00
Tom Rhodes
ce66ddb763 s/filesystem/file system/g as discussed on -developers 2002-08-21 18:11:48 +00:00
Kirk McKusick
1c85e6a35d This commit adds basic support for the UFS2 filesystem. The UFS2
filesystem expands the inode to 256 bytes to make space for 64-bit
block pointers. It also adds a file-creation time field, an ability
to use jumbo blocks per inode to allow extent like pointer density,
and space for extended attributes (up to twice the filesystem block
size worth of attributes, e.g., on a 16K filesystem, there is space
for 32K of attributes). UFS2 fully supports and runs existing UFS1
filesystems. New filesystems built using newfs can be built in either
UFS1 or UFS2 format using the -O option. In this commit UFS1 is
the default format, so if you want to build UFS2 format filesystems,
you must specify -O 2. This default will be changed to UFS2 when
UFS2 proves itself to be stable. In this commit the boot code for
reading UFS2 filesystems is not compiled (see /sys/boot/common/ufsread.c)
as there is insufficient space in the boot block. Once the size of the
boot block is increased, this code can be defined.

Things to note: the definition of SBSIZE has changed to SBLOCKSIZE.
The header file <ufs/ufs/dinode.h> must be included before
<ufs/ffs/fs.h> so as to get the definitions of ufs2_daddr_t and
ufs_lbn_t.

Still TODO:
Verify that the first level bootstraps work for all the architectures.
Convert the utility ffsinfo to understand UFS2 and test growfs.
Add support for the extended attribute storage. Update soft updates
to ensure integrity of extended attribute storage. Switch the
current extended attribute interfaces to use the extended attribute
storage. Add the extent like functionality (framework is there,
but is currently never used).

Sponsored by: DARPA & NAI Labs.
Reviewed by:	Poul-Henning Kamp <phk@freebsd.org>
2002-06-21 06:18:05 +00:00
Tom Rhodes
3468b317cb more file system > filesystem 2002-05-16 04:10:46 +00:00
Poul-Henning Kamp
75766e179d Sigh, more BBSIZE related breakage.
Sponsored by:	DARPA & NAI Labs.
2002-05-12 21:37:08 +00:00
Warner Losh
d476a036e2 o remove __P
o remove main prototype
2002-03-21 13:20:49 +00:00
Ian Dowse
47f07d95ba Don't require that the special/filesystem argument translates into
a block or character device; the rest of tunefs works just fine on
filesystem images in regular files. Instead, if getfsfile() failed
and if the specified filesystem is a directory then print a more
useful "unknown file system" error.

Also, _PATH_DEV already contains a trailing slash, so don't add
another one when constructing a device path, and use errx() instead
of err() in a case where errno is meangingless.
2001-09-30 14:57:08 +00:00
Kris Kennaway
55fd28c86b sprintf -> snprintf
Obtained from:	OpenBSD
MFC After:	1 week
2001-07-24 11:40:18 +00:00
Dima Dorfman
c33fa91f61 Constify, de-register-ify, and set WARNS=2.
Submitted by:	Mike Barcroft <mike@q9media.com>
2001-07-15 05:47:47 +00:00
Kirk McKusick
1c2665d807 Do not allow the soft updates flag to be set if the filesystem is dirty.
Because the kernel will allow the mounting of unclean filesystems when
the soft updates flag is set, it is important that only soft updates
style inconsistencies (missing blocks and inodes) be present. Otherwise
a panic may ensue. It is also important that the filesystem be in a clean
state when the soft updates flag is set because the background fsck uses
the fact that the flag is set to indicate that it is safe to run. If
background fsck encounters non-soft updates style inconsistencies, it
will exit with unexpected inconsistencies.
2001-04-13 23:54:49 +00:00
Kirk McKusick
a61ab64ac4 Directory layout preference improvements from Grigoriy Orlov <gluk@ptci.ru>.
His description of the problem and solution follow. My own tests show
speedups on typical filesystem intensive workloads of 5% to 12% which
is very impressive considering the small amount of code change involved.

------

  One day I noticed that some file operations run much faster on
small file systems then on big ones. I've looked at the ffs
algorithms, thought about them, and redesigned the dirpref algorithm.

  First I want to describe the results of my tests. These results are old
and I have improved the algorithm after these tests were done. Nevertheless
they show how big the perfomance speedup may be. I have done two file/directory
intensive tests on a two OpenBSD systems with old and new dirpref algorithm.
The first test is "tar -xzf ports.tar.gz", the second is "rm -rf ports".
The ports.tar.gz file is the ports collection from the OpenBSD 2.8 release.
It contains 6596 directories and 13868 files. The test systems are:

1. Celeron-450, 128Mb, two IDE drives, the system at wd0, file system for
   test is at wd1. Size of test file system is 8 Gb, number of cg=991,
   size of cg is 8m, block size = 8k, fragment size = 1k OpenBSD-current
   from Dec 2000 with BUFCACHEPERCENT=35

2. PIII-600, 128Mb, two IBM DTLA-307045 IDE drives at i815e, the system
   at wd0, file system for test is at wd1. Size of test file system is 40 Gb,
   number of cg=5324, size of cg is 8m, block size = 8k, fragment size = 1k
   OpenBSD-current from Dec 2000 with BUFCACHEPERCENT=50

You can get more info about the test systems and methods at:
http://www.ptci.ru/gluk/dirpref/old/dirpref.html

                              Test Results

             tar -xzf ports.tar.gz               rm -rf ports
  mode  old dirpref new dirpref speedup old dirprefnew dirpref speedup
                             First system
 normal     667         472      1.41       477        331       1.44
 async      285         144      1.98       130         14       9.29
 sync       768         616      1.25       477        334       1.43
 softdep    413         252      1.64       241         38       6.34
                             Second system
 normal     329         81       4.06       263.5       93.5     2.81
 async      302         25.7    11.75       112          2.26   49.56
 sync       281         57.0     4.93       263         90.5     2.9
 softdep    341         40.6     8.4        284          4.76   59.66

"old dirpref" and "new dirpref" columns give a test time in seconds.
speedup - speed increasement in times, ie. old dirpref / new dirpref.

------

Algorithm description

The old dirpref algorithm is described in comments:

/*
 * Find a cylinder to place a directory.
 *
 * The policy implemented by this algorithm is to select from
 * among those cylinder groups with above the average number of
 * free inodes, the one with the smallest number of directories.
 */

A new directory is allocated in a different cylinder groups than its
parent directory resulting in a directory tree that is spreaded across
all the cylinder groups. This spreading out results in a non-optimal
access to the directories and files. When we have a small filesystem
it is not a problem but when the filesystem is big then perfomance
degradation becomes very apparent.

What I mean by a big file system ?

  1. A big filesystem is a filesystem which occupy 20-30 or more percent
     of total drive space, i.e. first and last cylinder are physically
     located relatively far from each other.
  2. It has a relatively large number of cylinder groups, for example
     more cylinder groups than 50% of the buffers in the buffer cache.

The first results in long access times, while the second results in
many buffers being used by metadata operations. Such operations use
cylinder group blocks and on-disk inode blocks. The cylinder group
block (fs->fs_cblkno) contains struct cg, inode and block bit maps.
It is 2k in size for the default filesystem parameters. If new and
parent directories are located in different cylinder groups then the
system performs more input/output operations and uses more buffers.
On filesystems with many cylinder groups, lots of cache buffers are
used for metadata operations.

My solution for this problem is very simple. I allocate many directories
in one cylinder group. I also do some things, so that the new allocation
method does not cause excessive fragmentation and all directory inodes
will not be located at a location far from its file's inodes and data.
The algorithm is:
/*
 * Find a cylinder group to place a directory.
 *
 * The policy implemented by this algorithm is to allocate a
 * directory inode in the same cylinder group as its parent
 * directory, but also to reserve space for its files inodes
 * and data. Restrict the number of directories which may be
 * allocated one after another in the same cylinder group
 * without intervening allocation of files.
 *
 * If we allocate a first level directory then force allocation
 * in another cylinder group.
 */

  My early versions of dirpref give me a good results for a wide range of
file operations and different filesystem capacities except one case:
those applications that create their entire directory structure first
and only later fill this structure with files.

  My solution for such and similar cases is to limit a number of
directories which may be created one after another in the same cylinder
group without intervening file creations. For this purpose, I allocate
an array of counters at mount time. This array is linked to the superblock
fs->fs_contigdirs[cg]. Each time a directory is created the counter
increases and each time a file is created the counter decreases. A 60Gb
filesystem with 8mb/cg requires 10kb of memory for the counters array.

  The maxcontigdirs is a maximum number of directories which may be created
without an intervening file creation. I found in my tests that the best
performance occurs when I restrict the number of directories in one cylinder
group such that all its files may be located in the same cylinder group.
There may be some deterioration in performance if all the file inodes
are in the same cylinder group as its containing directory, but their
data partially resides in a different cylinder group. The maxcontigdirs
value is calculated to try to prevent this condition. Since there is
no way to know how many files and directories will be allocated later
I added two optimization parameters in superblock/tunefs. They are:

        int32_t  fs_avgfilesize;   /* expected average file size */
        int32_t  fs_avgfpdir;      /* expected # of files per directory */

These parameters have reasonable defaults but may be tweeked for special
uses of a filesystem. They are only necessary in rare cases like better
tuning a filesystem being used to store a squid cache.

I have been using this algorithm for about 3 months. I have done
a lot of testing on filesystems with different capacities, average
filesize, average number of files per directory, and so on. I think
this algorithm has no negative impact on filesystem perfomance. It
works better than the default one in all cases. The new dirpref
will greatly improve untarring/removing/coping of big directories,
decrease load on cvs servers and much more. The new dirpref doesn't
speedup a compilation process, but also doesn't slow it down.

Obtained from:	Grigoriy Orlov <gluk@ptci.ru>
2001-04-10 08:38:59 +00:00
Ben Smithurst
e50fa3d247 Fix 'tunefs -p'
Reviewed by:	sheldonh
2001-01-29 11:00:16 +00:00
Philippe Charnier
77edab906e The tunefs code assumed that the last argument was the device specification.
We need to parse the arguments first, then open the device (if
specified) and then apply the changes. This change will disallow the
(undocumented) use of multiple instances of the same argument on the
same command line for the sack of a better error message.

Other changes are:
1) the softupdates (-n) now issue a warning about remaining unchanged
2) the usage and man page is changed to specify "space | time" instead of
"optimization preference".

PR:		bin/23335
Submitted by:Mark Peek <mark@whistle.com>
2000-12-10 20:59:30 +00:00
Philippe Charnier
2af14b60ed Remove .Op when arg is required (special | filesystem). Document that at
least one flag is required and check this in the code. Make use of getopt(3).
Generalyze printing `... remains unchanged ...'.
2000-11-28 18:17:15 +00:00
Sheldon Hearn
060ac658cc Open the device read-only initially and re-open read-write if necessary
later.  This allows tunefs -p on mounted filesystems.

Side-effects:
	Use K&R prototypes.
	Use definitions from fcntl.h for the flags argument to open(2).

There are cosmetic differences between this and the submitted patch.

PR:		17143
Reported by:	Peter Edwards <peter.edwards@ireland.com>
Submitted by:	luoqi
2000-03-14 07:44:32 +00:00
Luoqi Chen
51003344a8 Remove unused #include and prototype declaration. 2000-01-30 05:24:55 +00:00