Commit Graph

114 Commits

Author SHA1 Message Date
Xin LI
5e8221100a Use %ju and cast to (uintmax_t) to avoid using PRI* macros.
Suggested by:	kevlo
2020-09-28 04:30:31 +00:00
Xin LI
6155466afb Fix build.
Pointy hat to:	delphij
MFC after:	3 days
2020-09-23 07:27:12 +00:00
Xin LI
4d2c300bde sbin/fsck_msdosfs: Fix an integer overflow on 32-bit platforms.
The purpose of checksize() is to verify that the referenced cluster
chain size matches the recorded file size (up to 2^32 - 1) in the
directory entry. We follow the cluster chain, then multiple the
cluster count by bytes per cluster to get the physical size, then
check it against the recorded size.

When a file is close to 4 GiB (between 4GiB - cluster size and 4GiB,
both non-inclusive), the product of cluster count and bytes per
cluster would be exactly 4 GiB. On 32-bit systems, because size_t
is 32-bit, this would wrap back to 0, which will cause the file be
truncated to 0.

Fix this by using 64-bit physicalSize instead.

This fix is inspired by an Android change request at
https://android-review.googlesource.com/c/platform/external/fsck_msdos/+/1428461

PR:		249533
Reviewed by:	kevlo
MFC after:	3 days
Differential Revision:	https://reviews.freebsd.org/D26524
2020-09-23 06:52:22 +00:00
Xin LI
85f4f80c42 Do not overflow when calculating file system size.
Reported by:		Hyeongseok Kim <hyeongseok kim lge com>
Reviewed by:		cem, Hyeongseok Kim
MFC after:		3 days
Differential Revision:	https://reviews.freebsd.org/D24603
2020-04-28 05:10:34 +00:00
Xin LI
401475f50c Fix a bug with dirty file system handling.
r356313 broke handling of dirty file system because we have restricted
the correction of "odd" byte sequences to checkfat(), and as a result
the dirty bit is never cleared.  The old fsck_msdosfs code would write
FAT twice to fix the dirty bit, which is also not ideal.

Fix this by introducing a new rountine, cleardirty() which will perform
the set of clean bit only, and use it in checkfilesys() if we thought
the file system was dirty.

Reviewed by:		cem, emaste
MFC after:		3 day
Differential Revision:	https://reviews.freebsd.org/D24581
2020-04-27 02:01:48 +00:00
Xin LI
9e4029ff49 Use humanize_number to format available and bad space sizes.
Reviewed by:		mckusick (earlier version)
MFC after:		2 weeks
Differential Revision:	https://reviews.freebsd.org/D23050
2020-02-10 04:16:41 +00:00
Xin LI
43f1940992 Diff reduction against NetBSD, no functional change.
MFC after:	1 week
2020-02-02 20:53:31 +00:00
Xin LI
d14a599d69 Tighten FAT checks and fix off-by-one error in corner case.
sbin/fsck_msdosfs/fat.c:
 - readfat:
    * Only truncate out-of-range cluster pointers (1, or greater than
      NumClusters but smaller than CLUST_RSRVD), as the current cluster
      may contain some data. We can't fix reserved cluster pointers at
      this pass, because we do no know the potential cluster preceding
      it.
    * Accept valid cluster for head bitmap. This is a no-op, and mainly
      to improve code readability, because the 1 is already handled in
      the previous else if block.
 - truncate_at: absorbed into checkchain.
 - checkchain: save the previous node we have traversed in case that we
   have a chain that ends with a special (>= CLUST_RSRVD) cluster, or is
   free. In these cases, we need to truncate at the cluster preceding the
   current cluster, as the current cluster contains a marker instead of
   a next pointer and can not be changed to CLUST_EOF (the else case can
   happen if the user answered "no" at some point in readfat()).
 - clearchain: correct the iterator for next cluster so that we don't
   stop after clearing the first cluster.
 - checklost: If checkchain() thinks the chain have no cluster, it
   doesn't make sense to reconnect it, so don't bother asking.

Reviewed by:	kevlo
MFC after:	24 days
X-MFC-With:	r356313
Differential Revision:	https://reviews.freebsd.org/D23065
2020-01-12 06:13:52 +00:00
Xin LI
d3dd66792b Correct off-by-two issue when determining FAT type.
In the code we used NumClusters as the upper (non-inclusive) boundary
of valid cluster number, so the actual value was 2 (CLUST_FIRST) more
than the real number of clusters. This causes a FAT16 media with
65524 clusters be treated as FAT32 and might affect FAT12 media with
4084 clusters as well.

To fix this, we increment NumClusters by CLUST_FIRST after the type
determination.

PR:		243179
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D23082
2020-01-11 17:41:20 +00:00
Xin LI
727d995c7d Apply typo fix from NetBSD, we have already applied all NetBSD changes so
update the NetBSD tag while I'm there.

MFC after:	2 weeks
2020-01-11 04:02:40 +00:00
Xin LI
ed0879d944 Require FAT to occupy at least one sector.
Obtained from:	Android https://r.android.com/1205830
MFC after:	3 days
2020-01-11 03:59:06 +00:00
Xin LI
9a67c570a9 fsck_msdosfs.8: document -M.
Reported by:	mckusick
Reviewed by:	mckusick, emaste, bcr
MFC after:	28 days
X-MFC-with:	r356313
Differential Revision:	https://reviews.freebsd.org/D23049
2020-01-07 04:33:14 +00:00
Xin LI
9708ba9f29 Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.

This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.

With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.

Address this by taking a different approach of validating the FAT.

The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry.  Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.

We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain.  This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case.  The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.

When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.

As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry.  For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.

sbin/fsck_msdosfs/boot.c:

 - Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
   number.
 - FAT32: check if root directory starts with a valid cluster number,
   moved from dir.c.  There is no point to proceed if the filesystem
   is already damaged beyond repair.

sbin/fsck_msdosfs/check.c:

 - Combine phase 1 and phase 2, now that the readfat() is able to
   detect cross chains.

sbin/fsck_msdosfs/dir.c:

 - Refactor code to use FAT accessor instead of accessing the internal
   representation of FAT table.
 - Make use of the cluster chain head bitmap.
 - Clarify and simplify directory entry check, remove unnecessary
   checks that are would be done at a later time (for example, whether
   the directory's second cluster is a valid one, which is examined
   more throughly in a later checkchain() and does not prevent us
   from proceeding further).

sbin/fsck_msdosfs/dosfs.h:

 - Remove internal representation of FAT table, which is replaced by
   the head bitmap that is opaque to other code.
 - Added a special CLUST_DEAD cluster type to indicate errors.

sbin/fsck_msdosfs/ext.h:

 - Added a flag that overrides mmap(2) setting.  The corresponding
   command line option, -M is intentionally undocumented as we do not
   expect users to need it.
 - Added accessors for FAT table and convert existing interface to use
   it.

sbin/fsck_msdosfs/fat.c:

 - Added head bitmap to represent whether a cluster is a head cluster.
 - Converted FAT internal representation to accessors.
 - Implemented a LRU cache for FAT32 when mmap(2) should not or can not
   be used.
 - _readfat: Attempt a mmap(2) and fall back to regular read for
   non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
   the cache with the first 4MiB of the entries.
 - readfat: Added support of head bitmap and use the population scan to
   detect bogus chains.
 - clusterdiff: removed, FATs are copied from the checked copy via
   writefat()/copyfat().
 - checkchain: calculates the length of a cluster chain and make sure
   that it ends with a valid EOF marker.
 - clearchain: follow and clear a chain and maintain the free cluster
   count.
 - checklost: convert to use head bitmap. At the end of all other scans,
   the remaining 'head' nodes are leaders of lost cluster chains.

sbin/fsck_msdosfs/fat.c:

 - Added a new -M option which is intentionally undocumented, to disable
   the use of mmap().

Reviewed by:	kevlo
MFC after:	1 month
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
Xin LI
73db93b889 Revert r356249 for now as it broke GCC builds. 2020-01-01 09:22:06 +00:00
Xin LI
b06cf1e44f Reduce memory footprint of fsck_msdosfs.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.

With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.

Address this by taking a different approach of validating the FAT.

The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry.  Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.

We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain.  This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case.  The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.

When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.

As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry.  For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.

sbin/fsck_msdosfs/boot.c:

 - Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
   number.
 - FAT32: check if root directory starts with a valid cluster number,
   moved from dir.c.  There is no point to proceed if the filesystem
   is already damaged beyond repair.

sbin/fsck_msdosfs/check.c:

 - Combine phase 1 and phase 2, now that the readfat() is able to
   detect cross chains.

sbin/fsck_msdosfs/dir.c:

 - Refactor code to use FAT accessor instead of accessing the internal
   representation of FAT table.
 - Make use of the cluster chain head bitmap.
 - Clarify and simplify directory entry check, remove unnecessary
   checks that are would be done at a later time (for example, whether
   the directory's second cluster is a valid one, which is examined
   more throughly in a later checkchain() and does not prevent us
   from proceeding further).

sbin/fsck_msdosfs/dosfs.h:

 - Remove internal representation of FAT table, which is replaced by
   the head bitmap that is opaque to other code.
 - Added a special CLUST_DEAD cluster type to indicate errors.

sbin/fsck_msdosfs/ext.h:

 - Added a flag that overrides mmap(2) setting.  The corresponding
   command line option, -M is intentionally undocumented as we do not
   expect users to need it.
 - Added accessors for FAT table and convert existing interface to use
   it.

sbin/fsck_msdosfs/fat.c:

 - Added head bitmap to represent whether a cluster is a head cluster.
 - Converted FAT internal representation to accessors.
 - Implemented a LRU cache for FAT32 when mmap(2) should not or can not
   be used.
 - _readfat: Attempt a mmap(2) and fall back to regular read for
   non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
   the cache with the first 4MiB of the entries.
 - readfat: Added support of head bitmap and use the population scan to
   detect bogus chains.
 - clusterdiff: removed, FATs are copied from the checked copy via
   writefat()/copyfat().
 - checkchain: calculates the length of a cluster chain and make sure
   that it ends with a valid EOF marker.
 - clearchain: follow and clear a chain and maintain the free cluster
   count.
 - checklost: convert to use head bitmap. At the end of all other scans,
   the remaining 'head' nodes are leaders of lost cluster chains.

sbin/fsck_msdosfs/fat.c:

 - Added a new -M option which is intentionally undocumented, to disable
   the use of mmap().

Reviewed by:	kevlo
MFC after:	1 month
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D22965
2020-01-01 07:43:08 +00:00
Xin LI
eb1c42c1f0 Avoid mixing cluster numbers and sector numbers. Makes code more readable.
Obtained from:	NetBSD
MFC after:	2 weeks
2019-09-15 19:41:54 +00:00
Xin LI
07282103f5 Correct overflow logic in fullpath().
Obtained from:	OpenBSD
MFC after:	3 days
2019-09-04 04:44:03 +00:00
Xin LI
cf955ebf6f Remove unneeded blank line. No functional change.
MFC after:	2 weeks
2019-08-30 06:06:12 +00:00
Xin LI
b770b08061 Comment boot block checks and perform additional sanity checks:
The following checks are now being enforced:

 - bpbBytesPerSec: only accept 512, 1024, 2048 and 4096.
 - bpbSecPerClust: only accept 1, 2, 4, 8, 16, 32, 64 and 128.
 - bpbResSectors: require non-zero.
 - bpbFATs: require non-zero.
 - bpbSectors: require zero for FAT32.
 - bpbFATsmall: require zero for FAT32.
 - bpbHugeSectors: require non-zero for FAT32.

Bail out if the BPB contained values that do not meet these requirements.

We also require FATsecs * FATsecs to not overflow 32-bit unsigned
integer.

Check for backup boot block was removed because the checker does not take
corrective action, and msdosfs driver ignores it too.
2019-08-26 06:41:17 +00:00
Xin LI
55d26365b1 Use calloc().
MFC after:	2 weeks
2019-08-19 05:24:42 +00:00
Xin LI
b9919ec960 Remove redundant check and wrong fix: fat.c checks already take care
about cluster chains.

Obtained from:	OpenBSD
MFC after:	2 weeks
2019-08-19 04:28:12 +00:00
Xin LI
0b253f2ba4 In ask(): override default option if any of alwaysyes/alwaysno/rdonly is
set.

MFC after:	2 weeks
2019-06-15 07:23:06 +00:00
Xin LI
1e3ffe6590 Blankspace. No actual code change.
MFC after:	2 weeks
2019-06-15 06:51:46 +00:00
Xin LI
bf6ed41692 Avoid out of boundary access when checking invalid long filenames.
Obtained from:	OpenBSD (dir.c,v 1.25)
MFC after:	3 days
2019-06-11 22:21:29 +00:00
Xin LI
669d33ef98 preen should work independently with alwaysyes and alwaysno.
PR:		238375
MFC after:	2 weeks
2019-06-07 01:38:51 +00:00
Xin LI
51a1c26a18 Don't increment cl after increment.
MFC after:	3 days
2019-06-04 07:02:20 +00:00
Xin LI
dad02d7d08 Don't cast result from malloc().
MFC after:	2 weeks
2019-04-15 06:33:05 +00:00
Xin LI
0e6eb62d96 Write string constant differently to improve readability.
Reported by:	rgrimes
Reviewed by:	rgrimes, emaste
MFC after:	13 days
Differential Revision:	https://reviews.freebsd.org/D19829
2019-04-06 03:42:15 +00:00
Xin LI
d325f3c243 Fix build.
MFC after:	2 weeks
X-MFC-With:	r345900
2019-04-05 02:37:10 +00:00
Xin LI
d78b6f1ee8 Implement checking of .' and ..' entries of subdirectory.
Reviewed by:		pfg
Obtained from:		Android b6ee08aadb%5E%21/
MFC after:		2 weeks
Differential Revision:	https://reviews.freebsd.org/D19824
2019-04-05 02:21:16 +00:00
Xin LI
fbf8a5605f Restore lfcl when LOSTDIR's chain was corrupted and overwritten
with invalid value.

Obtained from:	Android 4d6d6f8a36%5E%21/
MFC after:	2 weeks
2019-04-04 23:34:03 +00:00
Xin LI
2bf0ee64f9 Restore the ability of checking and fixing next free
cluster in FSINFO that was lost in r203872.

Obtained from:	NetBSD
MFC after:	2 weeks
2019-04-04 23:16:36 +00:00
Xin LI
f806eb62a4 Assert that q can't be NULL. 'empty' is always non-NULL when DIREMPTY
is set earlier.

MFC after:	1 month
2019-04-03 07:09:28 +00:00
Xin LI
9ace37176e Distinguish between lseek errors and read errores.
MFC after:	2 weeks
2019-03-28 18:20:47 +00:00
Xin LI
7f2b7ec93c Detect and handle invalid number of FATs
If the number of FATs field in the boot sector is zero, give
an appropriate error code.

Obtained from:	Android 6c29bbe8d5%5E%21/
MFC after:	2 weeks
2018-07-13 02:02:16 +00:00
Xin LI
48f3bd63b9 Revert Makefile@335696 that sneaked into the commit.
X-MFC with:	335696
MFC after:	2 weeks
2018-06-27 06:50:24 +00:00
Xin LI
6f0f107256 Detect exFAT filesystems and abort if found and tighten BPB sanity
check.

Obtained from:	Android https://android-review.googlesource.com/61827
MFC after:	2 weeks
2018-06-27 06:49:20 +00:00
Xin LI
f7a300540f Fix division by zero when reading boot block by postponing division
until it is necessary and after we validated bytes per sector is non-
zero.

Obtained from:	Android https://android-review.googlesource.com/c/platform/external/fsck_msdos/+/36362
MFC after:	2 weeks
2018-06-26 06:18:59 +00:00
Xin LI
b7d6282aae Don't bail out when we find primary and secondary bootblocks miscompare.
We do not have code to fix this situation, and the mismatch does not
prevent the kernel driver from consuming the file system, and some factory
formatted SD cards seem to have a garbage backup block.

This makes the code match to its comments (replacing pfatal with pwarn).

Inspired by:	NetBSD r1.13
Inspired by:	b47b16353f
MFC after:	2 weeks
2018-06-22 16:18:19 +00:00
Xin LI
3e855e9c21 Don't bail out from the check if readboot() returns !FSFATAL.
This can happen when the fsinfo signature is invalid, and the
user have choose to fix it, in which case the code would return
FSBOOTMOD (not FSOK but not FSFATAL either).

All other (fatal) cases would return FSFATAL.

Obtained from:	Android Open Source Project
Obtained from:	d8775a29ea
MFC after:	2 weeks
2018-04-30 05:57:55 +00:00
Xin LI
aa031ef4a3 Prevent OOB access on corrupted msdos directories.
Submitted by:	Veo Zhang <veo live com>
MFC after:	2 weeks
2017-11-30 08:28:17 +00:00
Pedro F. Giffuni
1de7b4b805 various: general adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.
2017-11-27 15:37:16 +00:00
Bryan Drewery
ea825d0274 DIRDEPS_BUILD: Update dependencies.
Sponsored by:	Dell EMC Isilon
2017-10-31 00:07:04 +00:00
Enji Cooper
22289a8c3d sbin: normalize paths using SRCTOP-relative paths or :H when possible
This simplifies make logic/output

MFC after:	1 month
Sponsored by:	Dell EMC Isilon
2017-03-04 11:33:01 +00:00
Sevan Janiyan
4cbf025741 Mention the version of NetBSD the utility originated from.
PR:		212476
Approved by:	bcr (mentor)
MFC after:	4 days
Differential Revision:	https://reviews.freebsd.org/D8105
2016-10-02 23:40:17 +00:00
Pedro F. Giffuni
dd3f00cbaf fsck_msdosfs: Adjust a check.
The on-disk FAT array does not include anything before CLUST_FIRST,
compensate in size check.

Obtained from:	NetBSD (CVS Rev. 1.20)
MFC after:	2 weeks
2016-05-04 22:27:22 +00:00
Marcelo Araujo
091ea4a091 Use macro MAX() from sys/param.h.
MFC after:	2 weeks.
2016-04-22 03:32:14 +00:00
Pedro F. Giffuni
f3858ada3e fsck_msdosfs: use NULL instead of zero for pointers. 2016-04-19 19:08:37 +00:00
Glen Barber
876d357fa7 MFH
Sponsored by:	The FreeBSD Foundation
2016-04-11 15:24:59 +00:00
Pedro F. Giffuni
33bed970fa fsck_msdosfs(8): Optimimize memsets
Obtained from:	NetBSD (bin/50908)
MFC after:	2 weeks
2016-04-06 15:28:26 +00:00