2017-11-27 15:37:16 +00:00
|
|
|
/*-
|
|
|
|
|
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
|
|
|
|
|
*
|
2001-07-09 10:35:18 +00:00
|
|
|
* Copyright (C) 1995, 1996, 1997 Wolfgang Solfrank
|
|
|
|
|
* Copyright (c) 1995 Martin Husemann
|
|
|
|
|
*
|
|
|
|
|
* Redistribution and use in source and binary forms, with or without
|
|
|
|
|
* modification, are permitted provided that the following conditions
|
|
|
|
|
* are met:
|
|
|
|
|
* 1. Redistributions of source code must retain the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer.
|
|
|
|
|
* 2. Redistributions in binary form must reproduce the above copyright
|
|
|
|
|
* notice, this list of conditions and the following disclaimer in the
|
|
|
|
|
* documentation and/or other materials provided with the distribution.
|
|
|
|
|
*
|
|
|
|
|
* THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND ANY EXPRESS OR
|
|
|
|
|
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
|
|
|
|
|
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
|
|
|
|
|
* IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY DIRECT, INDIRECT,
|
|
|
|
|
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
|
|
|
|
|
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
|
|
|
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
|
|
|
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
|
|
|
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
|
|
|
|
|
* THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
|
* $NetBSD: ext.h,v 1.6 2000/04/25 23:02:51 jdolecek Exp $
|
|
|
|
|
* $FreeBSD$
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
#ifndef EXT_H
|
2014-07-14 21:32:40 +00:00
|
|
|
#define EXT_H
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
#include <sys/types.h>
|
|
|
|
|
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
#include <stdbool.h>
|
|
|
|
|
|
2001-07-09 10:35:18 +00:00
|
|
|
#include "dosfs.h"
|
|
|
|
|
|
|
|
|
|
#define LOSTDIR "LOST.DIR"
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Options:
|
|
|
|
|
*/
|
|
|
|
|
extern int alwaysno; /* assume "no" for all questions */
|
|
|
|
|
extern int alwaysyes; /* assume "yes" for all questions */
|
|
|
|
|
extern int preen; /* we are preening */
|
|
|
|
|
extern int rdonly; /* device is opened read only (supersedes above) */
|
2004-02-05 15:47:46 +00:00
|
|
|
extern int skipclean; /* skip clean file systems if preening */
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
extern int allow_mmap; /* allow the use of mmap() */
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* function declarations
|
|
|
|
|
*/
|
2005-03-09 10:10:51 +00:00
|
|
|
int ask(int, const char *, ...) __printflike(2, 3);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
2003-12-27 06:44:32 +00:00
|
|
|
/*
|
|
|
|
|
* Check the dirty flag. If the file system is clean, then return 1.
|
|
|
|
|
* Otherwise, return 0 (this includes the case of FAT12 file systems --
|
|
|
|
|
* they have no dirty flag, so they must be assumed to be unclean).
|
|
|
|
|
*/
|
|
|
|
|
int checkdirty(int, struct bootblock *);
|
|
|
|
|
|
2001-07-09 10:35:18 +00:00
|
|
|
/*
|
2002-08-21 18:11:48 +00:00
|
|
|
* Check file system given as arg
|
2001-07-09 10:35:18 +00:00
|
|
|
*/
|
2002-03-20 22:57:10 +00:00
|
|
|
int checkfilesys(const char *);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Return values of various functions
|
|
|
|
|
*/
|
|
|
|
|
#define FSOK 0 /* Check was OK */
|
|
|
|
|
#define FSBOOTMOD 1 /* Boot block was modified */
|
|
|
|
|
#define FSDIRMOD 2 /* Some directory was modified */
|
|
|
|
|
#define FSFATMOD 4 /* The FAT was modified */
|
|
|
|
|
#define FSERROR 8 /* Some unrecovered error remains */
|
2010-02-14 12:30:30 +00:00
|
|
|
#define FSFATAL 16 /* Some unrecoverable error occurred */
|
2014-07-14 21:32:40 +00:00
|
|
|
#define FSDIRTY 32 /* File system is dirty */
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
/*
|
2010-02-14 12:30:30 +00:00
|
|
|
* read a boot block in a machine independent fashion and translate
|
2001-07-09 10:35:18 +00:00
|
|
|
* it into our struct bootblock.
|
|
|
|
|
*/
|
2002-03-20 22:57:10 +00:00
|
|
|
int readboot(int, struct bootblock *);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Correct the FSInfo block.
|
|
|
|
|
*/
|
2002-03-20 22:57:10 +00:00
|
|
|
int writefsinfo(int, struct bootblock *);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
/* Opaque type */
|
|
|
|
|
struct fat_descriptor;
|
Reduce memory footprint of fsck_msdosfs.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-01 07:43:08 +00:00
|
|
|
|
2020-04-27 02:01:48 +00:00
|
|
|
int cleardirty(struct fat_descriptor *);
|
|
|
|
|
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
void fat_clear_cl_head(struct fat_descriptor *, cl_t);
|
|
|
|
|
bool fat_is_cl_head(struct fat_descriptor *, cl_t);
|
|
|
|
|
|
|
|
|
|
cl_t fat_get_cl_next(struct fat_descriptor *, cl_t);
|
|
|
|
|
|
|
|
|
|
int fat_set_cl_next(struct fat_descriptor *, cl_t, cl_t);
|
|
|
|
|
|
|
|
|
|
cl_t fat_allocate_cluster(struct fat_descriptor *fat);
|
|
|
|
|
|
|
|
|
|
struct bootblock* fat_get_boot(struct fat_descriptor *);
|
|
|
|
|
int fat_get_fd(struct fat_descriptor *);
|
|
|
|
|
bool fat_is_valid_cl(struct fat_descriptor *, cl_t);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
/*
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
* Read the FAT 0 and return a pointer to the newly allocated
|
|
|
|
|
* descriptor of it.
|
2001-07-09 10:35:18 +00:00
|
|
|
*/
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
int readfat(int, struct bootblock *, struct fat_descriptor **);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Write back FAT entries
|
|
|
|
|
*/
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
int writefat(struct fat_descriptor *);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Read a directory
|
|
|
|
|
*/
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
int resetDosDirSection(struct fat_descriptor *);
|
2002-03-20 22:57:10 +00:00
|
|
|
void finishDosDirSection(void);
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
int handleDirTree(struct fat_descriptor *);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Cross-check routines run after everything is completely in memory
|
|
|
|
|
*/
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
int checkchain(struct fat_descriptor *, cl_t, size_t *);
|
|
|
|
|
|
2001-07-09 10:35:18 +00:00
|
|
|
/*
|
|
|
|
|
* Check for lost cluster chains
|
|
|
|
|
*/
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
int checklost(struct fat_descriptor *);
|
2001-07-09 10:35:18 +00:00
|
|
|
/*
|
|
|
|
|
* Try to reconnect a lost cluster chain
|
|
|
|
|
*/
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
int reconnect(struct fat_descriptor *, cl_t, size_t);
|
2002-03-20 22:57:10 +00:00
|
|
|
void finishlf(void);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Small helper functions
|
|
|
|
|
*/
|
|
|
|
|
/*
|
|
|
|
|
* Return the type of a reserved cluster as text
|
|
|
|
|
*/
|
2012-10-21 12:01:19 +00:00
|
|
|
const char *rsrvdcltype(cl_t);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Clear a cluster chain in a FAT
|
|
|
|
|
*/
|
Reduce memory footprint of fsck_msdosfs.
This is a re-apply r356249 with changes to make GCC happy.
This utility was initially written for FAT12/16, which were inherently
small. When FAT32 support was added, the old data structure and
algorithms remain used with minimal changes.
With growing size of FAT32 media, the current data structure that
requires 4 32-bit variables per each FAT32 table entry would consume up
to 4 GiB of RAM, which can be too big for systems with limited RAM
available.
Address this by taking a different approach of validating the FAT.
The FAT is essentially a set of linked lists of chains that was
referenced by directory entries, and the checker needs to make sure that
the linked chains of clusters do not have cross-linked chains, and every
chain were referenced by one and only one directory entry. Instead of
keeping track of the chain's 'head' cluster number, the size of the
chain, the used status of the chain and the "next" pointer which is
content of the FAT table, we create accessors for the FAT table data
for the "next" pointer, and keep only one bit to indicate if the
current cluster is a 'head' node of a cluster chain, in a bitmap.
We further overhaul the FAT checker to find out the possible head nodes
by excluding ones that are not (in other words, nodes that have some
other nodes claiming them as the next node) instead of marking the head
nodes for each node on the chain. This approach greatly reduced the
complexiety of computation from O(N^2) worst case, to an O(N) scan for
worst case. The file (cluster chain) length is not useful for the FAT
checker, so don't bother to calculate them in the FAT checker and
instead leave the task to the directory structure check, at which point
we would have non-crossed cluster chains, and we are guaranteed that
each cluster will be visited for at most one time.
When checking the directory structures, we use the head node indicator
to as the visited (used) flag: every cluster chain can only be
referenced by one directory entry, so we clear them when calculating
the length of the chain, and we can immediately tell if there are
anomalies in the directory entry.
As a result, the required RAM size is now 1 bit per each entry of
the FAT table, plus memory needed to hold the FAT table in memory,
instead of 16 bytes (=128 bits) per each entry. For FAT12 and FAT16,
we will load the whole FAT table into memory as they are smaller than
128KiB, and for FAT32, we first attempt to mmap() it into memory, and
when that fails, we would fall back to a simple LRU cache of 4 MiB of
RAM.
sbin/fsck_msdosfs/boot.c:
- Added additional sanity checks for valid FAT32/FAT16/FAT12 cluster
number.
- FAT32: check if root directory starts with a valid cluster number,
moved from dir.c. There is no point to proceed if the filesystem
is already damaged beyond repair.
sbin/fsck_msdosfs/check.c:
- Combine phase 1 and phase 2, now that the readfat() is able to
detect cross chains.
sbin/fsck_msdosfs/dir.c:
- Refactor code to use FAT accessor instead of accessing the internal
representation of FAT table.
- Make use of the cluster chain head bitmap.
- Clarify and simplify directory entry check, remove unnecessary
checks that are would be done at a later time (for example, whether
the directory's second cluster is a valid one, which is examined
more throughly in a later checkchain() and does not prevent us
from proceeding further).
sbin/fsck_msdosfs/dosfs.h:
- Remove internal representation of FAT table, which is replaced by
the head bitmap that is opaque to other code.
- Added a special CLUST_DEAD cluster type to indicate errors.
sbin/fsck_msdosfs/ext.h:
- Added a flag that overrides mmap(2) setting. The corresponding
command line option, -M is intentionally undocumented as we do not
expect users to need it.
- Added accessors for FAT table and convert existing interface to use
it.
sbin/fsck_msdosfs/fat.c:
- Added head bitmap to represent whether a cluster is a head cluster.
- Converted FAT internal representation to accessors.
- Implemented a LRU cache for FAT32 when mmap(2) should not or can not
be used.
- _readfat: Attempt a mmap(2) and fall back to regular read for
non-FAT32 file systems; use the LRU cache for FAT32 and prepopulate
the cache with the first 4MiB of the entries.
- readfat: Added support of head bitmap and use the population scan to
detect bogus chains.
- clusterdiff: removed, FATs are copied from the checked copy via
writefat()/copyfat().
- checkchain: calculates the length of a cluster chain and make sure
that it ends with a valid EOF marker.
- clearchain: follow and clear a chain and maintain the free cluster
count.
- checklost: convert to use head bitmap. At the end of all other scans,
the remaining 'head' nodes are leaders of lost cluster chains.
sbin/fsck_msdosfs/fat.c:
- Added a new -M option which is intentionally undocumented, to disable
the use of mmap().
Reviewed by: kevlo
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D22965
2020-01-03 00:31:48 +00:00
|
|
|
void clearchain(struct fat_descriptor *, cl_t);
|
2001-07-09 10:35:18 +00:00
|
|
|
|
|
|
|
|
#endif
|
