Traditionally, fsck is invoked before the filesystems are mounted
and all checks are done to completion at that time. If background
checking is available, fsck is invoked twice. It is first invoked
at the traditional time, before the filesystems are mounted, with
the -F flag to do checking on all the filesystems that cannot do
background checking. It is then invoked a second time, after the
system has completed going multiuser, with the -B flag to do checking
on all the filesystems that can do background checking. Unlike
the foreground checking, the background checking is started
asynchonously so that other system activity can proceed even on
the filesystems that are being checked.
At the moment, only the fast filesystem supports background checking.
To be able to do background checking, a filesystem must have been
running with soft updates, not have been marked as needing a
foreground check, and be mounted and writable when the background
check is to be done (i.e., not listed as `noauto' in /etc/fstab).
These changes are the final piece needed to support background
filesystem checking. They will not have any effect until you update
your /etc/rc to invoke fsck in its new mode of operation. I am
still playing around with exactly what those changes should be
and should be committing them later this week.
filesystem needs foreground checking (usually at boot time) or
can defer to background checking (after the system is up and running).
See the manual page, fsck_ffs(8), for details on the -F and -B options.
These options are primarily intended for use by the fsck front end.
All output is directed to stdout so that the output is coherent
when redirected to a file or a pipe. Unify the code with the fsck
front end that allows either a device or a mount point to be
specified as the argument to be checked.
always look up -network and -mask addresses numerically before
trying getnetbyname(). Without this, we may end up attempting DNS
queries on silly names such as "127.0.0.0.my-domain.com". See the
commit log from revisions 1.21 and 1.20 for further details.
removes the last path component until the mount() succeeds. However,
the code never checks if it has passed the mountpoint, so in some
cases where the mount() never succeeds, it can end up applying the
flags from a mounted filesystem to the underlying one.
Add a sanity check to the code which removes the last path component:
test that the fsid associated with the new path is the same as that
of the old one.
PR: bin/7872
a number of assumptions related to the parsing of options in
/etc/exports, and missed a few necessary new error checks.
The main problems related to netmasks: an IPv6 network address
missing a netmask would result in the filesystem being exported to
the whole IPv6 world, non-continuous netmasks would be made continuous
without any warnings, and nothing prevented you specifying an IPv4
mask with an IPv6 address.
This change addresses these issues. As a side-effect we now store
netmasks in sockaddr structs (this matches the kernel interface,
and is closer to the way it used to be). Add a flag OP_HAVEMASK to
keep track of whether or not we have successfully got a mask from
any source. Replace some mask-related helper functions with versions
that use the sockaddr-based masks.
Also tidy up get_net() and fix the code that interprets IPv4 partial
networks such as "127.1" as network rather than host addresses.
Properly zero out some structures that were ending up partially
containing junk from the stack, fix a few formatting issues, and
add a comment noting some assumptions about export arguments.
would call malloc, stdio and other library functions from the signal
handler which is not safe due to reentrancy problems.
Instead, add a simple handler that just sets a flag, and call the
more complex function from main() when necessary. Unfortunately to
be able to check this flag, we must expand the svc_run() call, but
the RPC library makes that relatively easy to do.
- Remove some horrible code that faked a "struct addrinfo" to be
later passed to freeaddrinfo(). Instead, add a new group type
"GT_DEFAULT" used to denote that the filesystem is exported to the
world, and treat this case separately.
- Don't clear the AI_CANONNAME flag in a struct addrinfo returned
by getaddrinfo. There's still a bit more struct addrinfo abuse
left in here.
- Simplify do_mount() slightly by using an addrinfo pointer to keep
track of the current address.
- Revert del_mlist() to its pre-tirpc prototype. Unlike NetBSD's version,
ours lets the caller generate any syslog() messages, so that it
can include the service name in the message.
- Initialise a few local variables to clarify the logic and avoid some
compiler warnings.
- Remove a few unused functions and local variables, and fix some
whitespace issues.
- Reinstate the logic for avoiding duplicate host entries that got
removed accidentally in revision 1.41 (added in r1.5). This bit
was submitted in a slightly different form by Thomas Quinot.
Submitted by: Martin Blapp <mb@imp.ch>,
Thomas Quinot <quinot@inf.enst.fr>
PR: bin/26148
1) Set the FS_NEEDSFSCK flag when unexpected problems are encountered.
2) Clear the FS_NEEDSFSCK flag after a successful foreground cleanup.
3) Refuse to run in background when the FS_NEEDSFSCK flag is set.
4) Avoid taking and removing a snapshot when the filesystem is already clean.
5) Properly implement the force cleaning (-f) flag when in preen mode.
Note that you need to have revision 1.21 (date: 2001/04/14 05:26:28) of
fs.h installed in <ufs/ffs/fs.h> defining FS_NEEDSFSCK for this to compile.
Because the kernel will allow the mounting of unclean filesystems when
the soft updates flag is set, it is important that only soft updates
style inconsistencies (missing blocks and inodes) be present. Otherwise
a panic may ensue. It is also important that the filesystem be in a clean
state when the soft updates flag is set because the background fsck uses
the fact that the flag is set to indicate that it is safe to run. If
background fsck encounters non-soft updates style inconsistencies, it
will exit with unexpected inconsistencies.
not -tag. Instead, put a period after the error messages to aide
those using dumb terminals not capable of properly displaying markup.
Requested by: ru
the ability to use a preprocessor, use the -q (quiet) flag when reading
from a file). The source used is from ipfw.
Clean up exit codes while I am here.
KAME has been informed and plans on integrating these patches into their
own source as well.
number of issues:
- Fix background mounts; these were broken in revision 1.40.
- Don't give up before trying all addresses returned by getaddrinfo().
- Use protocol-independent routines where possible.
- Improve error reporting for RPC errors.
- In non-background mode, give up after trying all protocols once.
- Use daemon(3) instead of rolling our own version.
- Never go ahead with the mount() syscall until we have received
a reply from the remote nfsd; this is especially important with
non-interruptible mounts, as otherwise a mistyped command might
require a reboot to correct.
Reviewed by: alfred, Martin Blapp <mb@imp.ch>
His description of the problem and solution follow. My own tests show
speedups on typical filesystem intensive workloads of 5% to 12% which
is very impressive considering the small amount of code change involved.
------
One day I noticed that some file operations run much faster on
small file systems then on big ones. I've looked at the ffs
algorithms, thought about them, and redesigned the dirpref algorithm.
First I want to describe the results of my tests. These results are old
and I have improved the algorithm after these tests were done. Nevertheless
they show how big the perfomance speedup may be. I have done two file/directory
intensive tests on a two OpenBSD systems with old and new dirpref algorithm.
The first test is "tar -xzf ports.tar.gz", the second is "rm -rf ports".
The ports.tar.gz file is the ports collection from the OpenBSD 2.8 release.
It contains 6596 directories and 13868 files. The test systems are:
1. Celeron-450, 128Mb, two IDE drives, the system at wd0, file system for
test is at wd1. Size of test file system is 8 Gb, number of cg=991,
size of cg is 8m, block size = 8k, fragment size = 1k OpenBSD-current
from Dec 2000 with BUFCACHEPERCENT=35
2. PIII-600, 128Mb, two IBM DTLA-307045 IDE drives at i815e, the system
at wd0, file system for test is at wd1. Size of test file system is 40 Gb,
number of cg=5324, size of cg is 8m, block size = 8k, fragment size = 1k
OpenBSD-current from Dec 2000 with BUFCACHEPERCENT=50
You can get more info about the test systems and methods at:
http://www.ptci.ru/gluk/dirpref/old/dirpref.html
Test Results
tar -xzf ports.tar.gz rm -rf ports
mode old dirpref new dirpref speedup old dirprefnew dirpref speedup
First system
normal 667 472 1.41 477 331 1.44
async 285 144 1.98 130 14 9.29
sync 768 616 1.25 477 334 1.43
softdep 413 252 1.64 241 38 6.34
Second system
normal 329 81 4.06 263.5 93.5 2.81
async 302 25.7 11.75 112 2.26 49.56
sync 281 57.0 4.93 263 90.5 2.9
softdep 341 40.6 8.4 284 4.76 59.66
"old dirpref" and "new dirpref" columns give a test time in seconds.
speedup - speed increasement in times, ie. old dirpref / new dirpref.
------
Algorithm description
The old dirpref algorithm is described in comments:
/*
* Find a cylinder to place a directory.
*
* The policy implemented by this algorithm is to select from
* among those cylinder groups with above the average number of
* free inodes, the one with the smallest number of directories.
*/
A new directory is allocated in a different cylinder groups than its
parent directory resulting in a directory tree that is spreaded across
all the cylinder groups. This spreading out results in a non-optimal
access to the directories and files. When we have a small filesystem
it is not a problem but when the filesystem is big then perfomance
degradation becomes very apparent.
What I mean by a big file system ?
1. A big filesystem is a filesystem which occupy 20-30 or more percent
of total drive space, i.e. first and last cylinder are physically
located relatively far from each other.
2. It has a relatively large number of cylinder groups, for example
more cylinder groups than 50% of the buffers in the buffer cache.
The first results in long access times, while the second results in
many buffers being used by metadata operations. Such operations use
cylinder group blocks and on-disk inode blocks. The cylinder group
block (fs->fs_cblkno) contains struct cg, inode and block bit maps.
It is 2k in size for the default filesystem parameters. If new and
parent directories are located in different cylinder groups then the
system performs more input/output operations and uses more buffers.
On filesystems with many cylinder groups, lots of cache buffers are
used for metadata operations.
My solution for this problem is very simple. I allocate many directories
in one cylinder group. I also do some things, so that the new allocation
method does not cause excessive fragmentation and all directory inodes
will not be located at a location far from its file's inodes and data.
The algorithm is:
/*
* Find a cylinder group to place a directory.
*
* The policy implemented by this algorithm is to allocate a
* directory inode in the same cylinder group as its parent
* directory, but also to reserve space for its files inodes
* and data. Restrict the number of directories which may be
* allocated one after another in the same cylinder group
* without intervening allocation of files.
*
* If we allocate a first level directory then force allocation
* in another cylinder group.
*/
My early versions of dirpref give me a good results for a wide range of
file operations and different filesystem capacities except one case:
those applications that create their entire directory structure first
and only later fill this structure with files.
My solution for such and similar cases is to limit a number of
directories which may be created one after another in the same cylinder
group without intervening file creations. For this purpose, I allocate
an array of counters at mount time. This array is linked to the superblock
fs->fs_contigdirs[cg]. Each time a directory is created the counter
increases and each time a file is created the counter decreases. A 60Gb
filesystem with 8mb/cg requires 10kb of memory for the counters array.
The maxcontigdirs is a maximum number of directories which may be created
without an intervening file creation. I found in my tests that the best
performance occurs when I restrict the number of directories in one cylinder
group such that all its files may be located in the same cylinder group.
There may be some deterioration in performance if all the file inodes
are in the same cylinder group as its containing directory, but their
data partially resides in a different cylinder group. The maxcontigdirs
value is calculated to try to prevent this condition. Since there is
no way to know how many files and directories will be allocated later
I added two optimization parameters in superblock/tunefs. They are:
int32_t fs_avgfilesize; /* expected average file size */
int32_t fs_avgfpdir; /* expected # of files per directory */
These parameters have reasonable defaults but may be tweeked for special
uses of a filesystem. They are only necessary in rare cases like better
tuning a filesystem being used to store a squid cache.
I have been using this algorithm for about 3 months. I have done
a lot of testing on filesystems with different capacities, average
filesize, average number of files per directory, and so on. I think
this algorithm has no negative impact on filesystem perfomance. It
works better than the default one in all cases. The new dirpref
will greatly improve untarring/removing/coping of big directories,
decrease load on cvs servers and much more. The new dirpref doesn't
speedup a compilation process, but also doesn't slow it down.
Obtained from: Grigoriy Orlov <gluk@ptci.ru>
[I first added this functionality, and thought to check prior art. Seeing
OpenBSD had already done this, I changed my addition to reduce the diffs
between the two and went with their option letter.]
Obtained from: OpenBSD
of pingnfsserver(). The pingnfsport() function is now called everytime.
If we don't get RPC_SUCCESS or RPC_PROGVERSMISMATCH back, there's
something wrong with the NFS server and we just exit.
- Fix cfs mount on IPv4-only machines
- Fixed the looping when we did not run background mode.
- Fixed a getnameinfo() call with uninitialized adress.
This is a NetBSD bug I didn't notified :-( Thanks Ian !
- Added some #ifdef NFSKERB
- Removed some unused variables.
- Fixed idention
- Remove unnecessary ":" in openlog ident
Submitted by: Martin Blapp <mb@imp.ch>
Reviewed by: Ian Dowse <iedowse@maths.tcd.ie>
<QUOTE>
Overflow of cg_niblk causes spurious "SUMMARY INFORMATION BAD" warnings
prompts in fsck_ffs. cg_niblk has type int16_t, but it is (ab)used
to hold the number of inodes per group (it has nothing to do with the
number of inode blocks in the cg as claimed in ffs/fs.h), and there
can be lots of inodes per group. E.g.,
newfs -i 8192 -b 16384 -f 16384 -c 255 ...
gives 0xA900 inodes/group. Assignment of this to cg_niblk gives
-0x5700. This is mostly harmless (ufs/ffs doesn't use cg_niblk, and
bug-for-bug compatible overflow occurs in newfs), but in
fsck_ffs/pass5.c, negative values of cg_niblk limit cg_irotor to 0
and the summary info appears to be bad.
</QUOTE>
Submitted by: Bruce Evans <bde@zeta.org.au>
Obtained from: Kirk McKusick <mckusick@mckusick.com>
Consider this /etc/fstab:
# Device Mountpoint FStype Options Dump Pass#
/dev/ad1s1b none swap sw 0 0
/dev/ad0s1b none swap sw 0 0
/dev/ad0s1a / ufs rw 1 1
/dev/ad0s1e /home ufs rw 2 2
/dev/ad1s1e /tmp ufs rw 2 2
/dev/ad1s1f /usr ufs rw 2 2
/dev/ccd0c /syv ufs rw 2 11
proc /proc procfs rw 0 0
ccd0c is striped over /dev/ad0f and /dev/ad1g
Without this pass, fsck in preen mode will check ad0s1a first,
and then issue three processes in parallel:
One process doing ad0s1e
One process doing ad1s1e and ad1s1f
One process doing ccd0c
There is no way to tell it that ccd0c overlaps ad0 and ad1.
With the patch, it will do it this way:
pass 2:
One process doing ad0s1e
One process doing ad1s1e and ad1s1f
and when they are complete:
pass 11:
One process doing ccd0c
This is much faster and more sane.
Valid pass numbers are anything from 1 to INTMAX-1.
I retired the '-l' option which tried to allow people to do
something like this, but which didn't work and which complicated
the code an awful lot.
If one is trying to dump or repair an ill system, give the user a fighting
chance. Refusing to operate w/o a very non-critical file (feature) is
just plain stupid.
Some of the major changes include:
- The SCSI error handling portion of cam_periph_error() has
been broken out into a number of subfunctions to better
modularize the code that handles the hierarchy of SCSI errors.
As a result, the code is now much easier to read.
- String handling and error printing has been significantly
revamped. We now use sbufs to do string formatting instead
of using printfs (for the kernel) and snprintf/strncat (for
userland) as before.
There is a new catchall error printing routine,
cam_error_print() and its string-based counterpart,
cam_error_string() that allow the kernel and userland
applications to pass in a CCB and have errors printed out
properly, whether or not they're SCSI errors. Among other
things, this helped eliminate a fair amount of duplicate code
in camcontrol.
We now print out more information than before, including
the CAM status and SCSI status and the error recovery action
taken to remedy the problem.
- sbufs are now available in userland, via libsbuf. This
change was necessary since most of the error printing code
is shared between libcam and the kernel.
- A new transfer settings interface is included in this checkin.
This code is #ifdef'ed out, and is primarily intended to aid
discussion with HBA driver authors on the final form the
interface should take. There is example code in the ahc(4)
driver that implements the HBA driver side of the new
interface. The new transfer settings code won't be enabled
until we're ready to switch all HBA drivers over to the new
interface.
src/Makefile.inc1,
lib/Makefile: Add libsbuf. It must be built before libcam,
since libcam uses sbuf routines.
libcam/Makefile: libcam now depends on libsbuf.
libsbuf/Makefile: Add a makefile for libsbuf. This pulls in the
sbuf sources from sys/kern.
bsd.libnames.mk: Add LIBSBUF.
camcontrol/Makefile: Add -lsbuf. Since camcontrol is statically
linked, we can't depend on the dynamic linker
to pull in libsbuf.
camcontrol.c: Use cam_error_print() instead of checking for
CAM_SCSI_STATUS_ERROR on every failed CCB.
sbuf.9: Change the prototypes for sbuf_cat() and
sbuf_cpy() so that the source string is now a
const char *. This is more in line wth the
standard system string functions, and helps
eliminate warnings when dealing with a const
source buffer.
Fix a typo.
cam.c: Add description strings for the various CAM
error status values, as well as routines to
look up those strings.
Add new cam_error_string() and
cam_error_print() routines for userland and
the kernel.
cam.h: Add a new CAM flag, CAM_RETRY_SELTO.
Add enumerated types for the various options
available with cam_error_print() and
cam_error_string().
cam_ccb.h: Add new transfer negotiation structures/types.
Change inq_len in the ccb_getdev structure to
be "reserved". This field has never been
filled in, and will be removed when we next
bump the CAM version.
cam_debug.h: Fix typo.
cam_periph.c: Modularize cam_periph_error(). The SCSI error
handling part of cam_periph_error() is now
in camperiphscsistatuserror() and
camperiphscsisenseerror().
In cam_periph_lock(), increase the reference
count on the periph while we wait for our lock
attempt to succeed so that the periph won't go
away while we're sleeping.
cam_xpt.c: Add new transfer negotiation code. (ifdefed
out)
Add a new function, xpt_path_string(). This
is a string/sbuf analog to xpt_print_path().
scsi_all.c: Revamp string handing and error printing code.
We now use sbufs for much of the string
formatting code. More of that code is shared
between userland the kernel.
scsi_all.h: Get rid of SS_TURSTART, it wasn't terribly
useful in the first place.
Add a new error action, SS_REQSENSE. (Send a
request sense and then retry the command.)
This is useful when the controller hasn't
performed autosense for some reason.
Change the default actions around a bit.
scsi_cd.c,
scsi_da.c,
scsi_pt.c,
scsi_ses.c: SF_RETRY_SELTO -> CAM_RETRY_SELTO. Selection
timeouts shouldn't be covered by a sense flag.
scsi_pass.[ch]: SF_RETRY_SELTO -> CAM_RETRY_SELTO.
Get rid of the last vestiges of a read/write
interface.
libkern/bsearch.c,
sys/libkern.h,
conf/files: Add bsearch.c, which is needed for some of the
new table lookup routines.
aic7xxx_freebsd.c: Define AHC_NEW_TRAN_SETTINGS if
CAM_NEW_TRAN_CODE is defined.
sbuf.h,
subr_sbuf.c: Add the appropriate #ifdefs so sbufs can
compile and run in userland.
Change sbuf_printf() to use vsnprintf()
instead of kvprintf(), which is only available
in the kernel.
Change the source string for sbuf_cpy() and
sbuf_cat() to be a const char *.
Add __BEGIN_DECLS and __END_DECLS around
function prototypes since they're now exported
to userland.
kdump/mkioctls: Include stdio.h before cam.h since cam.h now
includes a function with a FILE * argument.
Submitted by: gibbs (mostly)
Reviewed by: jdp, marcel (libsbuf makefile changes)
Reviewed by: des (sbuf changes)
Reviewed by: ken
So bump the default from `16' to `22', which is the largest value allowed
with the current default block size. This change increases the the
group size from 32MB/g to 44MB/g on a 4GB SCSI disk.
