Generally the first argument in calloc is supposed to stand for a count
and the second for a size. Try to make that consistent. While here,
attempt to make some use of the overflow detection capability in
calloc(3).
mkimg has had a number of functional additions after the last time the
version was incremented. Do so now, to r292082's commit date, so that
users can determine what is supported.
Reviewed by: marcel
Approved by: re (gjb)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6882
mkimg(1) uses a swap file to back input file chunks. When the output file
is being written out, blocks of the swap file are mapped and their contents
copied. This causes the backing VM pages to enter the active queue, and when
the output file is large relative to system memory (as is generally the
case), can result in a shortfall of inactive memory. This causes the
pagedaemon to aggressively scan the active queue and swap out process
memory in an attempt to meet the shortfall. Because mkimg's input files
are typically the intermediate result of some build process, there's no
need to push them all through the active queue. Use madvise(2) to indicate
that the backing pages may be reclaimed in preference to active pages. In
the case of the swap file, these pages will be freed as soon as mkimg
exits anyway.
When using mkimg on a desktop-class system with large amounts of dirty
process memory, this change substantially improves mkimg runtime and
reduces swap usage.
Reviewed by: marcel
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D6654
after r298107
Summary of changes:
- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)
Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.
MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division
This is not properly respecting WITHOUT or ARCH dependencies in target/.
Doing so requires a massive effort to rework targets/ to do so. A
better approach will be to either include the SUBDIR Makefiles directly
and map to DIRDEPS or just dynamically lookup the SUBDIR. These lose
the benefit of having a userland/lib, userland/libexec, etc, though and
results in a massive package. The current implementation of targets/ is
very unmaintainable.
Currently rescue/rescue and sys/modules are still not connected.
Sponsored by: EMC / Isilon Storage Division
netbsd-tests.test.mk (r289151)
- Eliminate explicit OBJTOP/SRCTOP setting
- Convert all ad hoc NetBSD test integration over to netbsd-tests.test.mk
- Remove unnecessary TESTSDIR setting
- Use SRCTOP where possible for clarity
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Divison
is taken to match the geometry and only when the geometry is max'd
out, is the actual recorded size taken.
Note that qemu has the same logic for the fixed VHD format. However
that is known to conflict with Microsoft Azure, where the recorded
size of the image is what counts.
Pointed out by: gjb@
The image is not accepted for provisioning otherwise. Bump the
VHD creator tool version and the version of mkimg to signify our
success in provisioning.
Note that this also imapcts the dynamic VHD images.
Tested by: gjb@
of megabytes. This is on top of having the image rounded to the
matching geometry of the image size.
By rounding up to the next MB after rounding to the geometry, we
lost idempotency. Subsequent calls to resize the image will keep
increasing the image size.
Tested by: gjb@
multiple of the cylinder size. This is what qemu-img seems to
be doing. Make sure to handle boundary cases where increasing
the image size by 1 cyclinder's worth would also result in a
change of geometry.
a capcity is given, no partitions are required. When no partitions are
given, no scheme needs to be specified either. This makes it possible
to create an entirely empty disk image. To add an empty partitioning
table, specify the scheme.
Bump the version to 20150222.
number of clusters it occupies. It's not the number of entries in the table,
as it is for the L1 cluster table.
For small images, the two are the same. With the unit tests based on small
images, this change has therefore no effect on the unit test. For larger
images (like the FreeBSD 10.1-RELEASE image), this gives a discrepancy that
actually shows up when running "qemu-img check".
Bump the version number of mkimg.
While here, fix a white-space bug.
MFC after: 1 week
with 128K of random data and truncated to 800K can have SEEK_DATA return -1
when given an offset of 128K. On UFS, the SEEK_DATA returns 800K (the size
of the file). SEEK_HOLE on ZFS seems to behave the same as UFS.
To handle this, map -1 to the size of the file (`end') when lseek returns
this for either SEEK_HOLE or SEEK_DATA. When sparse files are not supported
by the file system both `hole' and `data' will now be equal to `end' and we
will treat the entire file as data. This way, the -1 return for SEEK_DATA
on ZFS will end up doing the right thing.
Reported by: gjb@
MFC after: 3 days
cylinder, head and track numbers. Return ~0U for these values when
mkimg wasn't given both -T and -H (i.e. no geometry) or the cylinder
would be larger than the provided maximum.
Use mkimgs_chs() for the EBR, MBR and PC98 schemes to fill in the
appropriate fields. Make sure to use a "rounded" size so that the
partition is always a multiple of the track size. We reserved the
room for it in the metadata callback so that's a valid thing to
do.
Bump the mkimg version number.
While doing that again: have mkimg.o depend on the Makefile so that
a version change triggers a rebuild as needed.
that keeps track of a particular region of the image. In particular the
image_data() function needs to return to the caller whether a region
contains data or is all zeroes. This required reading the region from
the temporary file and comparing the bytes. When image_data() is used
multiple times for the same region, this will get painful fast.
With a chunk describing a region of the image, we now also have a way
to refer to the image provided on the command line. This means we don't
need to copy the image into a temporary file. We just keep track of the
file descriptor and offset within the source file on a per-chunk basis.
For streams (pipes, sockets, fifos, etc) we now use the temporary file
as a swap file. We read from the input file and create a chunk of type
"zeroes" for each sequence of zeroes that's a multiple of the sector
size. Otherwise, we allocte from the swap file, mmap(2) it, read into
the mmap(2)'d memory and create a chunk representing data.
For regular files, we use SEEK_HOLE and SEEK_DATA to handle sparse files
eficiently and create a chunk of type zeroes for holes and a chunk of
type data for data regions. For data regions, we still compare the bytes
we read to handle differences between a file system's block size and our
sector size.
After reading all files, image_write() is used by schemes to scribble in
the reserved sectors. Since this never amounts to much, keep this data
in memory in chunks of exactly 1 sector.
The output image is created by looking using the chunk list to find the
data and write it out to the output file. For chunks of type "zeroes"
we prefer to seek, but fall back to writing zeroes to handle pipes.
For chunks of type "file" and "memoty" we simply write.
The net effect of this is that for reasonably large images the execution
time drops from 1-2 minutes to 10-20 seconds. A typical speedup is about
5 to 8 times, depending on partition sizes, output format whether in
input files are sparse or not.
Bump version to 20141001.
options. Bump the version number to 20140927.
While here, use explicit fputc() calls to skip a line in the output.
This to avoid having to hunt for extra '\n' characters in the printf
format strings.
MFC after: 1 week
Relnotes: yes
--version print the version of mkimg and also whether it's
64- or 32-bit.
--formats list the supported output formats separated by space.
--schemes list the supported partitioning schemes separated by
space.
Inspired by a patch from: gjb@
MFC after: 1 week
Relnotes: yes
We have a different ordering for the RC block(s) and L2 tables.
This is expected to be a non-issue, because everything is found
through file offsets in the corresponding RC table and L1 table.
Files that grow organically have RC blocks and L2 tables scattered
all over the place anyway.
The reason for the difference is that mkimg needs to be able to
write to a pipe. We can't seek forward and backward to fill in
the bits in non-sequential order.
variable was assigned the image offset in bytes and not in blocks
(i.e. sectors). This had image_data() return FALSE, which meant that
we didn't assign a cluster when we needed and also meant that we
didn't write parts of the L2 table when we should have. The result
being that the actual data clusters were written at the wrong offset.
Improve support for QCOW version 2. We're having the right layout
and even know how many refcnt blocks we need. All we need to do is
populate the refcnt blocks for every cluster we write and allocate
a cluster when we need a new refcnt block. The allocation part is
tricky in that it'll interleave with the assignment of clusters to
L2 tables and data. Since version 2 is not quite done, keep it
compiled out for now.
trying to get the test name right, failed, gave up and used a sequence
number instead. When I realized it wasn't because of the number of
underscores in the name that I really started to think. I didn't have
braces around the variable names ...
Thus: test_1 is now called apm_1x1_512_qcow, which gives you all you
need to run mkimg by hand.
Dumb-ass: marcel
to the baseline. Since we don't run gzip with the -n option, the output
of gzip varies for identical result files if and when they are created
at different time. Ouch...
Rather than add -n and commit a 600K+ diff for the changes to all the .uu
files, it's less of a churn to uudecode and gunzip the baseline file and
compare that to the new result file to determine if the baseline file
needs to be updated.
This way, "atf-sh mkimg.sh rebase" can be run as many times as people like
and a subsequent "svn status" will not show unnecessary diffs.
