These were found by the Undefined Behaviour GsoC project at NetBSD:
Do not change signedness bit with left shift.
While there avoid signed integer overflow.
Address both issues with using unsigned type.
msdosfs_fat.c:512:42, left shift of 1 by 31 places cannot be represented
in type 'int'
msdosfs_fat.c:521:44, left shift of 1 by 31 places cannot be represented
in type 'int'
msdosfs_fat.c:744:14, left shift of 1 by 31 places cannot be represented
in type 'int'
msdosfs_fat.c:744:24, signed integer overflow: -2147483648 - 1 cannot be
represented in type 'int [20]'
msdosfs_fat.c:840:13, left shift of 1 by 31 places cannot be represented
in type 'int'
msdosfs_fat.c:840:36, signed integer overflow: -2147483648 - 1 cannot be
represented in type 'int [20]'
Detected with micro-UBSan in the user mode.
Hinted from: NetBSD (CVS 1.33)
MFC after: 2 weeks
Differenctial Revision: https://reviews.freebsd.org/D16615
(https://svnweb.freebsd.org/base?view=revision&revision=304232)
converting clrbuf() (which clears the entire buffer) to vfs_bio_clrbuf()
(which clears only the new pages that have been added to the buffer).
Failure to properly remove pages from the buffer cache can make
pages that appear not to need clearing to actually have bad random
data in them. See for example base r304232
(https://svnweb.freebsd.org/base?view=revision&revision=304232)
which noted the need to set B_INVAL and B_NOCACHE as well as clear
the B_CACHE flag before calling brelse() to release the buffer.
Rather than trying to find all the incomplete brelse() calls, it
is simpler, though more slightly expensive, to simply clear the
entire buffer when it is newly allocated.
PR: 213507
Submitted by: Damjan Jovanovic
Reviewed by: kib
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.
The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
FAT specification requires that for valid FAT, FAT cluster 0 has a
specific value derived from the BPB media descriptor. The lowest
(little-endian) byte must be equal to bpb.bpbMedia, other bits in the
cluster number must be all 1's. Implement the check to reduce the
chance of the randomly corrupted FAT to pass the mount attempt.
Submitted by: Siva Mahadevan <smahadevan@freebsdfoundation.org>
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D12124
This somewhat simplifies use of msdosfs code in userland (for makefs),
reduces diffs with NetBSD and is standard C as of C89.
Reviewed by: imp
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D11014
General cleanup, for diff reduction with NetBSD and future use by FAT
support in makefs.
Submitted by: Siva Mahadevan <smahadevan@freebsdfoundation.org>
Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10821
volume limits. In particular:
- Assert that usemap_alloc() and usemap_free() cluster number argument
is valid.
- In chainlength(), return 0 if cluster start is after the max cluster.
- In chainlength(), cut the calculated cluster chain length at the max
cluster.
- For true paranoia, after the pm_inusemap is calculated in
fillinusemap(), reset all bits in the array for clusters after the
max cluster, as in-use.
Reported and tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
There are a number of msdosfs improvements in NetBSD that may be worth
bringing over, and this reduces noise in the comparison.
Differential Revision: https://reviews.freebsd.org/D1466
Reviewed by: kib
Sponsored by: The FreeBSD Foundation
is excessive. Postpone the flush of the fsinfo to VFS_SYNC(),
remembering the need for update with the flag MSDOSFS_FSIMOD, stored
in pm_flags.
FAT32 specification describes both FSI_Free_Count and FSI_Nxt_Free as
the advisory hints, not requiring them to be correct.
Based on the patch from bde, modified by me.
Reviewed by: bde
MFC after: 2 weeks
track the MNT_SYNCHRONOUS flag. It is set to the latter at mount time
but not updated by MNT_UPDATE.
Use MNT_SYNCHRONOUS to decide to write the FAT updates syncrhonously.
Submitted by: bde
MFC after: 1 week
The plan is to use vnode lock to protect denode and fat cache,
and having separate lock for block use map.
Change the check and return on impossible condition into KASSERT().
Tested by: pho
MFC after: 3 weeks
replaced by file relative sector numbers as the buffer block number when
zero-padding a file during extension. Revert the change, it causes wrong
blocks filled with zeroes on seeking beyond end of file.
PR: kern/47628
Submitted by: tegge
MFC after: 3 days
All active fields in fsi are advisory/optional, so we shouldn't do
extra work to make them valid at all times, but instead we write to
the fsi too often (we still do), and we searched for a free cluster
for fsinxtfree too often.
This commit just removes the whole search and its results, so that we
write out our in-core copy of fsinxtfree instead of writing a "fixed"
copy and clobbering our in-core copy. This saves fixing 3 bugs:
- off-by-1 error for the end of the search, resulting in fsinxtfree
not actually being adjusted iff only the last cluster is free.
- missing adjustment when no clusters are free.
- off-by-many error for the start of the search. Starting the search
at 0 instead of at (the in-core copy of) fsinxtfree did more than
defeat the reasons for existence of fsinxtfree. fsinxtfree exists
mainly to avoid having to start at 0 for just the first search per
mount, but has the side effect of reducing bias towards allocating
near cluster 0. The bias would normally only be generated by the
first search per mount (if fsinxtfree is not supported), but since
we also adjusted the in-core copy of fsinxtfree here, we were doing
extra work to maximize the bias.
Approved by: re (kensmith)
message explained why the size is 1 sector, but the code used a
size of 1 cluster.
I/o sizes larger than necessary may cause serious coherency problems
in the buffer cache. Here I think there were only minor efficiency
problems, since a too-large fsinfo buffer could only get far enough
to overlap buffers for the same vnode (the device vnode), so mappings
are coherent at the page level although not at the buffer level, and
the former is probably enough due to our limited use of the fsinfo
buffer.
Approved by: re (kensmith)
In msdosfs_read(), mainly reorder the main loop to the same order as in
ffs_read().
In msdosfs_write() and extendfile(), use vfs_bio_clrbuf() instead of
clrbuf(). I think this just just a bogus optimization, but ffs always
does it and msdosfs already did it in one place, and it is what I've
tested.
In msdosfs_write(), merge good bits from a comment in ffs_write(), and
fix 1 style bug.
In the main comment for msdosfs_pcbmap(), improve wording and catch
up with 13 years of changes in the function. This comment belongs in
VOP_BMAP.9 but that doesn't exist.
In msdosfs_bmap(), return EFBIG if the requested cluster number is out
of bounds instead of blindly truncating it, and fix many style bugs.
Approved by: re (hrs)
part of fixing msdosfs for large sector sizes. One of the fixed bugs
was fatal for large sector sizes.
1. The fsinfo block has size 512, but it was misunderstood and declared
as having size 1024, with nothing in the second 512 bytes except a
signature at the end. The second 512 bytes actually normally (if
the file system was created by Windows) consist of a second boot
sector which is normally (in WinXP) empty except for a signature --
the normal layout is one boot sector, one fsinfo sector, another
boot sector, then these 3 sectors duplicated. However, other
layouts are valid. newfs_msdos produces a valid layout with one
boot sector, one fsinfo sector, then these 2 sectors duplicated.
The signature check for the extra part of the fsinfo was thus
normally checking the signature in either the second boot sector
or the first boot sector in the copy, and thus accidentally
succeeding. The extra signature check would just fail for weirder
layouts with 512-byte sectors, and for normal layouts with any other
sector size.
Remove the extra bytes and the extra signature check.
2. Old versions did i/o to the fsinfo block using size 1024, with the
second half only used for the extra signature check on read. This
was harmless for sector size 512, and worked accidentally for sector
size 1024. The i/o just failed for larger sector sizes.
The version being fixed did i/o to the fsinfo block using size
fsi_size(pmp) = (1024 << ((pmp)->pm_BlkPerSec >> 2)). This
expression makes no sense. It happens to work for sector small
sector sizes, but for sector size 32K it gives the preposterous
value of 64M and thus causes panics. A sector size of 32768 is
necessary for at least some DVD-RW's (where the minimum write size
is 32768 although the minimum read size is 2048).
Now that the size of the fsinfo block is 512, it always fits in
one sector so there is no need for a macro to express it. Just
use the sector size where the old code uses 1024.
Approved by: re (kensmith)
Approved by: nyan (several years ago for a different version of (2))
of the the first cluster in a file (and, if the allocation cannot be
continued contiguously, for subsequent clusters in a file) was randomized
in an attempt to leave space for contiguous allocation of subsequent
clusters in each file when there are multiple writers. This reduced
internal fragmentation by a few percent, but it increased external
fragmentation by up to a few thousand percent.
Use simple sequential allocation instead. Actually maintain the fsinfo
sequence index for this. The read and write of this index from/to
disk still have many non-critical bugs, but we now write an index that
has something to do with our allocations instead of being modified
garbage. If there is no fsinfo on the disk, then we maintain the index
internally and don't go near the bugs for writing it.
Allocating the first free cluster gives a layout that is almost as good
(better in some cases), but takes too much CPU if the FAT is large and
the first free cluster is not near the beginning.
The effect of this change for untar and tar of a slightly reduced copy
of /usr/src on a new file system was:
Before (msdosfs 4K-clusters):
untar: 459.57 real untar from cached file (actually a pipe)
tar: 342.50 real tar from uncached tree to /dev/zero
Before (ffs2 soft updates 4K-blocks 4K-frags)
untar: 39.18 real
tar: 29.94 real
Before (ffs2 soft updates 16K-blocks 2K-frags)
untar: 31.35 real
tar: 18.30 real
After (msdosfs 4K-clusters):
untar 54.83 real
tar 16.18 real
All of these times can be improved further.
With multiple concurrent writers or readers (especially readers), the
improvement is smaller, but I couldn't find any case where it is
negative. 342 seconds for tarring up about 342 MB on a ~47MB/S partition
is just hard to unimprove on. (This operation would take about 7.3
seconds with reasonably localized allocation and perfect read-ahead.)
However, for active file systems, 342 seconds is closer to normal than
the 16+ seconds above or the 11 seconds with other changes (best I've
measured -- won easily by msdosfs!). E.g., my active /usr/src on ffs1
is quite old and fragmented, so reading to prepare for the above
benchmark takes about 6 times longer than reading back the fresh copies
of it.
Approved by: re (kensmith)
from just before extending a file. This has the desired effect
of keeping the write speed constant. And yes, that helps a lot
copying large files always at full speed now, and I have seen
improvements using benchmarks/bonnie.
Stolen from: NetBSD
Reviewed by: bde
Fix a panic that occurred when trying to traverse a corrupt msdosfs
filesystem. With this particular corruption, the code in pcbmap()
would compute an offset into an array that was way out of bounds,
so check the bounds before trying to access and return an error if
the offset would be out of bounds.
Submitted by: Xin LI
(msdosfs uses normal 8-char indentation almost everywhere else),
too-long lines, and minor English usage errors. The verbose formal
comment before the new function is still abnormal.
- Define one flag GB_LOCK_NOWAIT that tells getblk() to pass the LK_NOWAIT
flag to the initial BUF_LOCK(). This will eventually be used in cases
were we want to use a buffer only if it is not currently in use.
- Convert all consumers of the getblk() api to use this extra parameter.
Reviwed by: arch
Not objected to by: mckusick
the bio and buffer structures to have daddr64_t bio_pblkno,
b_blkno, and b_lblkno fields which allows access to disks
larger than a Terabyte in size. This change also requires
that the VOP_BMAP vnode operation accept and return daddr64_t
blocks. This delta should not affect system operation in
any way. It merely sets up the necessary interfaces to allow
the development of disk drivers that work with these larger
disk block addresses. It also allows for the development of
UFS2 which will use 64-bit block addresses.
