Commit Graph

25 Commits

Author SHA1 Message Date
Warner Losh
4d846d260e spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with:		pfg
MFC After:		3 days
Sponsored by:		Netflix
2023-05-12 10:44:03 -06:00
Xin LI
85f4f80c42 Do not overflow when calculating file system size.
Reported by:		Hyeongseok Kim <hyeongseok kim lge com>
Reviewed by:		cem, Hyeongseok Kim
MFC after:		3 days
Differential Revision:	https://reviews.freebsd.org/D24603
2020-04-28 05:10:34 +00:00
Xin LI
401475f50c Fix a bug with dirty file system handling.
r356313 broke handling of dirty file system because we have restricted
the correction of "odd" byte sequences to checkfat(), and as a result
the dirty bit is never cleared.  The old fsck_msdosfs code would write
FAT twice to fix the dirty bit, which is also not ideal.

Fix this by introducing a new rountine, cleardirty() which will perform
the set of clean bit only, and use it in checkfilesys() if we thought
the file system was dirty.

Reviewed by:		cem, emaste
MFC after:		3 day
Differential Revision:	https://reviews.freebsd.org/D24581
2020-04-27 02:01:48 +00:00
Xin LI
9e4029ff49 Use humanize_number to format available and bad space sizes.
Reviewed by:		mckusick (earlier version)
MFC after:		2 weeks
Differential Revision:	https://reviews.freebsd.org/D23050
2020-02-10 04:16:41 +00:00
Xin LI
9708ba9f29 Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.

This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.

With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.

Address this by taking a different approach of validating the FAT.

The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry.  Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.

We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain.  This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case.  The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.

When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.

As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry.  For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.

sbin/fsck_msdosfs/boot.c:

 - Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
   number.
 - FAT32: check if root directory starts with a valid cluster number,
   moved from dir.c.  There is no point to proceed if the filesystem
   is already damaged beyond repair.

sbin/fsck_msdosfs/check.c:

 - Combine phase 1 and phase 2, now that the readfat() is able to
   detect cross chains.

sbin/fsck_msdosfs/dir.c:

 - Refactor code to use FAT accessor instead of accessing the internal
   representation of FAT table.
 - Make use of the cluster chain head bitmap.
 - Clarify and simplify directory entry check, remove unnecessary
   checks that are would be done at a later time (for example, whether
   the directory's second cluster is a valid one, which is examined
   more throughly in a later checkchain() and does not prevent us
   from proceeding further).

sbin/fsck_msdosfs/dosfs.h:

 - Remove internal representation of FAT table, which is replaced by
   the head bitmap that is opaque to other code.
 - Added a special CLUST_DEAD cluster type to indicate errors.

sbin/fsck_msdosfs/ext.h:

 - Added a flag that overrides mmap(2) setting.  The corresponding
   command line option, -M is intentionally undocumented as we do not
   expect users to need it.
 - Added accessors for FAT table and convert existing interface to use
   it.

sbin/fsck_msdosfs/fat.c:

 - Added head bitmap to represent whether a cluster is a head cluster.
 - Converted FAT internal representation to accessors.
 - Implemented a LRU cache for FAT32 when mmap(2) should not or can not
   be used.
 - _readfat: Attempt a mmap(2) and fall back to regular read for
   non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
   the cache with the first 4MiB of the entries.
 - readfat: Added support of head bitmap and use the population scan to
   detect bogus chains.
 - clusterdiff: removed, FATs are copied from the checked copy via
   writefat()/copyfat().
 - checkchain: calculates the length of a cluster chain and make sure
   that it ends with a valid EOF marker.
 - clearchain: follow and clear a chain and maintain the free cluster
   count.
 - checklost: convert to use head bitmap. At the end of all other scans,
   the remaining 'head' nodes are leaders of lost cluster chains.

sbin/fsck_msdosfs/fat.c:

 - Added a new -M option which is intentionally undocumented, to disable
   the use of mmap().

Reviewed by:	kevlo
MFC after:	1 month
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
Xin LI
73db93b889 Revert r356249 for now as it broke GCC builds. 2020-01-01 09:22:06 +00:00
Xin LI
b06cf1e44f Reduce memory footprint of fsck_msdosfs.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.

With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.

Address this by taking a different approach of validating the FAT.

The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry.  Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.

We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain.  This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case.  The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.

When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.

As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry.  For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.

sbin/fsck_msdosfs/boot.c:

 - Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
   number.
 - FAT32: check if root directory starts with a valid cluster number,
   moved from dir.c.  There is no point to proceed if the filesystem
   is already damaged beyond repair.

sbin/fsck_msdosfs/check.c:

 - Combine phase 1 and phase 2, now that the readfat() is able to
   detect cross chains.

sbin/fsck_msdosfs/dir.c:

 - Refactor code to use FAT accessor instead of accessing the internal
   representation of FAT table.
 - Make use of the cluster chain head bitmap.
 - Clarify and simplify directory entry check, remove unnecessary
   checks that are would be done at a later time (for example, whether
   the directory's second cluster is a valid one, which is examined
   more throughly in a later checkchain() and does not prevent us
   from proceeding further).

sbin/fsck_msdosfs/dosfs.h:

 - Remove internal representation of FAT table, which is replaced by
   the head bitmap that is opaque to other code.
 - Added a special CLUST_DEAD cluster type to indicate errors.

sbin/fsck_msdosfs/ext.h:

 - Added a flag that overrides mmap(2) setting.  The corresponding
   command line option, -M is intentionally undocumented as we do not
   expect users to need it.
 - Added accessors for FAT table and convert existing interface to use
   it.

sbin/fsck_msdosfs/fat.c:

 - Added head bitmap to represent whether a cluster is a head cluster.
 - Converted FAT internal representation to accessors.
 - Implemented a LRU cache for FAT32 when mmap(2) should not or can not
   be used.
 - _readfat: Attempt a mmap(2) and fall back to regular read for
   non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
   the cache with the first 4MiB of the entries.
 - readfat: Added support of head bitmap and use the population scan to
   detect bogus chains.
 - clusterdiff: removed, FATs are copied from the checked copy via
   writefat()/copyfat().
 - checkchain: calculates the length of a cluster chain and make sure
   that it ends with a valid EOF marker.
 - clearchain: follow and clear a chain and maintain the free cluster
   count.
 - checklost: convert to use head bitmap. At the end of all other scans,
   the remaining 'head' nodes are leaders of lost cluster chains.

sbin/fsck_msdosfs/fat.c:

 - Added a new -M option which is intentionally undocumented, to disable
   the use of mmap().

Reviewed by:	kevlo
MFC after:	1 month
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D22965
2020-01-01 07:43:08 +00:00
Xin LI
3e855e9c21 Don't bail out from the check if readboot() returns !FSFATAL.
This can happen when the fsinfo signature is invalid, and the
user have choose to fix it, in which case the code would return
FSBOOTMOD (not FSOK but not FSFATAL either).

All other (fatal) cases would return FSFATAL.

Obtained from:	Android Open Source Project
Obtained from:	d8775a29ea
MFC after:	2 weeks
2018-04-30 05:57:55 +00:00
Pedro F. Giffuni
1de7b4b805 various: general adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.
2017-11-27 15:37:16 +00:00
Pedro F. Giffuni
f07b643c5c newfs_msdosfs: Respect FSFIXFAT
Fix some whitespace issues while here.

Obtained from:	NetBSD (rev. 1.9)
MFC after:	3 days
2014-07-14 21:32:40 +00:00
Jaakko Heinonen
2923ae8a85 Print a newline after the error message.
PR:		bin/168447
Submitted by:	Boris Kochergin
2012-11-03 09:18:37 +00:00
Ulrich Spörlein
6cf357bc6c sbin/fsck: s/perror/perr/ to avoid shadowing
- rename some other vars too
- merge NetBSD license changes

Obtained from:	NetBSD
PR:		bin/139802
Reviewed by:	ed
2012-10-21 12:01:11 +00:00
Brian Somers
48790afa7b Fix some style(9), although there's a lot more issues here.
Fix some casting errors.

PR:		142384
Submitted by:	giffunip at tutopia dot com
Obtained from:	NetBSD
MFC after:	3 weeks
2010-06-20 09:40:54 +00:00
Konstantin Belousov
75fb535394 Rename fields to match better the msdosfs headers. This work is still
incomplete as some info doesn't really belong to the structs where it is
defined.

Submitted by:	Pedro F. Giffuni <giffunip tutopia com>
Reviewed by:	bde
MFC after:	2 weeks
2010-02-14 12:31:28 +00:00
Konstantin Belousov
6069db97f6 Bug fixes from NetBSD
- fix sign-compare issues.
- ANSIfy a couple of functions.
- Remove more duplicate #includes.
- Memory leak found by Coverity on NetBSD.

Submitted by:	Pedro F. Giffuni <giffunip tutopia com>
Reviewed by:	bde
MFC after:	2 weeks
2010-02-14 12:30:30 +00:00
Konstantin Belousov
a1882ff2a2 License changes from NetBSD.
Move to 2 clause license, approved by Wolfgang Solfrank.

Submitted by:	Pedro F. Giffuni <giffunip tutopia com>
MFC after:	2 weeks
2010-02-14 12:28:43 +00:00
Bruce Evans
ae62d94069 Fixed operation of -f to match its documentation and fsck_ffs. It
has now has no effect except in combination with -p, and plain fsck
checks all file systems instead of skipping clean ones for msdosfs
only.

Renamed the force flag to skipclean and inverted its logic as in
fsck_ffs.
2004-02-05 15:47:46 +00:00
Bruce Evans
a270f31ebd Prepare to fix checkdirty() by moving it from check.c to fat.c. It is
identical to a subset of readfat(), so it belongs near readfat() if not
in it.
2004-02-05 06:32:16 +00:00
Bruce Evans
7e1cecfd0f Oops, highly non-KNF indentation is normal for large expressions in
this program.  Gnu indentation is used for these.  Redo the fix for
the large expression at the end of the previous commit to give gnu
indentation.  The original version was gnuish but had 9 bogus extra
characters of indentation in its continuation lines, perfect tab
lossage on every line, and other bugs.

The previous commit log should have claimed to fix style bugs in the
previous-1 commit (1.5), not the forced null previous commit (1.6).
2003-12-28 01:30:31 +00:00
Bruce Evans
f7bf3122d9 Fixed some style bugs in previous commit (mainly highly non-KNF indentation). 2003-12-27 06:21:44 +00:00
Tom Rhodes
cede1f563c Make msdosfs support the dirty flag in FAT16 and FAT32.
Enable lockf support.

PR:		55861
Submitted by:	Jun Su <junsu@m-net.arbornet.org> (original version)
Reviewed by:	make universe
2003-12-26 17:19:19 +00:00
Tom Rhodes
565e3e6567 In check.c:
Avoid shadowing declarations.
Avoid compairing signed and unsigned types.
2003-10-30 09:08:09 +00:00
Tom Rhodes
ce66ddb763 s/filesystem/file system/g as discussed on -developers 2002-08-21 18:11:48 +00:00
Tom Rhodes
3468b317cb more file system > filesystem 2002-05-16 04:10:46 +00:00
David E. O'Brien
0121b42aca Add fsck_msdosfs.
Obtained from:	NetBSD
2001-07-09 10:35:18 +00:00