Commit Graph

119 Commits

Author SHA1 Message Date
gonzo
d907bbbffd Fix invalid VHDX generation for image larger than 4Gb
- Part of BAT payload location was lost due to invalid
    BAT entry encoding type (32 bits instead of 64 bits)
- The sequence of PB/SB entries in BAT was broken due to
    off-by-one index check. It worked for smaller than
    4Gb because there were no SB entries in BAT.

MFC after:	1 day
2020-07-13 02:24:31 +00:00
bapt
a095da0a47 Revert r362466
Such change should not have happen without prior discussion and review.

With hat:	transitioning core
2020-06-22 07:46:24 +00:00
hselasky
2333998f66 Improve wording to be more precise and clear.
No functional change intended.

s/Master Boot/Main Boot/ (also called MBR)

MFC after:	1 week
Sponsored by:	Mellanox Technologies
2020-06-21 13:34:08 +00:00
0mp
7d94c999fc Use Fl instead of Ar for long flags
Also, bump date after r361935.

MFC after:	1 week
2020-06-08 21:11:34 +00:00
gonzo
aefd5f515c Add VHDX support to mkimg(1)
VHDX is the successor of Microsoft's VHD file format. It increases
maximum capacity of the virtual drive to 64TB and introduces features
to better handle power/system failures.

VHDX is the required format for 2nd generation Hyper-V VMs.

Reviewed by:	marcel
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D25184
2020-06-08 20:28:32 +00:00
arichardson
9b0805591f Allow bootstrapping mkimg on macOS/Linux
On these systems the (u)int64_t typedefs will not be implicitly defined by the
previous includes, so include <stdint.h> in the header that uses uint64_t.

Reviewed By:	brooks
Differential Revision: https://reviews.freebsd.org/D23202
2020-01-16 14:14:55 +00:00
marcel
d78c98bbbe Don't compress and uuencode the "hexdump -C" output files. Just
save them with the $FreeBSD$ tag prepended.  Changes to these
files are now a lot easier to comprehend, which makes diffs also
reviewable.
2019-03-09 02:03:07 +00:00
marcel
e54089c483 Round # partitions up to fill the last GPT table sector
Set the number of partitions entries in the GPT header to a
multiple of the number of entries that fit in a sector.

PR:		236238
Reviewed by:	imp
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D19465
2019-03-05 22:55:33 +00:00
delphij
1cceceb8e2 Remove unneeded headers.
MFC after:	1 month
2019-01-06 20:39:23 +00:00
benno
10246e837a Fix a conditional that got mucked up.
Sponsored by:	iXsystems, Inc.
2018-04-12 17:16:13 +00:00
benno
19e7651c74 Add the ability to specify absolute and relative offsets to size partitions.
To create hybrid boot media we want to specify a partition at a known location.
This extends the syntax of size partitions to include an optional offset that
can be absolute or relative. It also introduces validation to make sure that
this hasn't resulted in overlapping partitions. I haven't added this to the
file and process partition specifications yet but the mechanics are designed
such that if someone comes up with a good way of specifying the offset it
will be fairly easy to add in.

Reviewed by:	imp
Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D14916
2018-04-12 15:47:47 +00:00
bdrewery
a598c4b809 DIRDEPS_BUILD: Update dependencies.
Sponsored by:	Dell EMC Isilon
2017-10-31 00:07:04 +00:00
ngie
d26727d972 Add HAS_TESTS to all Makefiles that are currently using the
`SUBDIR.${MK_TESTS}+= tests` idiom.

This is a follow up to r321912.
2017-08-02 08:50:42 +00:00
ngie
734d081ed1 MFhead@r321912 2017-08-02 08:38:36 +00:00
ngie
d2a33e87cf Update the usr.bin/mkimg golden test output files after ^/head@r319125
^/head@r319125 changed the location of the backup pmbr, requiring the
output files to be regenerated, since they're binary disk dumps.

The output files were regenerated with "make rebase"--fixed in
^/head@r319294.

MFC with:	r319125, r319294
PR:		219673
Sponsored by:	Dell EMC Isilon
2017-05-31 08:01:12 +00:00
ngie
9cffbb118e Fix "make rebase" after ^/head@r315776
"make rebase" can be used for rebasing the output files from mkimg
after making a change to mkimg. This will come in handy soon, per
bug 219673.

MFC after:	3 days
Sponsored by:	Dell EMC Isilon
2017-05-31 07:49:49 +00:00
ngie
4968926298 Formalize the dependent/dependency relationship for <foo>.gz.uu vs <foo>
This helps ensure that the output files are regenerated if the input files
change, after the output files have been created.

MFC after:	3 days
Sponsored by:	Dell EMC Isilon
2017-05-31 07:42:14 +00:00
manu
9dc3b7708f mkimg: Correct an off by one error in the PMBR size
The PMBR last sector should be number of sector - 1 (As stated in UEFI Spec
2.6 page 118 table 17).
This fixes warning printed by linux tools like parted or fdisk.

Sponsored by:	Gandi.net
2017-05-29 12:51:02 +00:00
manu
bf54eea21a mkimg: Add -C argument to specify maximum capacity
Add a -C option to specify a maximum capacity for the final image file.
It is useful to control the size of the generated image for sdcard or
when we will add dynamic size partition.

Add --capacity which is a shorthand to define min and max capacity at
the same time.

Reviewed by:	bapt, marcel, wblock (manpages)
Sponsored by:	Gandi.net
Differential Revision:	https://reviews.freebsd.org/D10509
2017-05-10 09:36:34 +00:00
ngie
5ad4f59bfb Rename tests from <foo> to <foo>_test to match the FreeBSD test suite
naming scheme

usr.bin/diff/diff_test was renamed to usr.bin/diff/netbsd_diff_test
to avoid collisions with the renamed FreeBSD test.

MFC after:	1 week
Sponsored by:	Dell EMC Isilon
2017-03-23 03:28:24 +00:00
pfg
bc68fe9602 mkimg(1): let calloc(3) do the multiplication.
Reviewed by:	marcel
MFC after:	1 week
2017-03-13 20:14:07 +00:00
sjg
6b4d12ca9b Allow building mkimg as cross-tool
For linux the mmap offset must also be page aligned, and we
need to disable macros like __FBSDID()

Change the linux osdep_uuidgen() to use more portable gettimeofday().

Reviewed by: marcel
2017-03-03 01:56:55 +00:00
bapt
bce2f5d2c8 Add missing section in manpage reference
Reported by:	make manlint
MFC after:	2 days
2017-02-11 23:41:38 +00:00
nyan
259480b6de Remove pc98 support completely.
I thank all developers and contributors for pc98.

Relnotes:	yes
2017-01-28 02:22:15 +00:00
imp
5f1361a7ee Add a new flag to mkimg (-a num) to specify the active partition for
those partitioning schemes that have this concept. Implement it as an
override for mbr's setting 0x80 in the flags for the first partition
when we have boot code.

Differential Revision: https://reviews.freebsd.org/D4403
2016-10-18 05:43:12 +00:00
marcel
a31300a1ec o Provide a private definition for UUIDs (mkimg_uuid_t) because
UUIDs are not portable.
 o  Move mkimg_uuid() to a new file and merge both gpt_uuid_enc()
    and vhd_uuid_enc() into a single mkimg_uuid_enc() that lives
    in the same file.
 o  Move the OS-specific implementation of generating a UUID to
    osdep_uuidgen() and provide the implementations for FreeBSD,
    macOS and Linux.
 o  Expect the partitioning scheme headers to be found by having
    a search to the directory in which the headers live. This
    avoids conflicts on non-FreeBSD machines.
2016-10-18 01:55:07 +00:00
marcel
3b59134ffe Switch to using the portable partition scheme headers. 2016-10-16 02:55:52 +00:00
marcel
2e6ae9742c Replace OFF_MAX with INT64_MAX. The former is defined on Linux. 2016-10-03 04:00:30 +00:00
marcel
5e84416ac9 Prefer <stdint.h> over <sys/types.h>. While here remove redundant
inclusion of <sys/queue.h>.

Move the inclusion of the disk partitioning headers out of order
and inbetween standard headers and local header. They will change
in a subsequent commit.
2016-10-03 02:37:28 +00:00
marcel
f258aa5cde Replace STAILQ with TAILQ. TAILQs are portable enough that they can
be used on both macOS and Linux. STAILQs are not. In particular,
STAILQ_LAST does not next on Linux. Since neither STAILQ_FOREACH_SAFE
nor TAILQ_FOREACH_SAFE exist on Linux, replace its use with a regular
TAILQ_FOREACH. The _SAFE variant was only used for having the next
pointer in a local variable.
2016-10-03 01:46:47 +00:00
marcel
abd401527f Portability changes:
1.  macOS nor Linux have MAP_NOCORE nor MAP_NOSYNC. Define as 0.
2.  macOS doesn't have SEEK_DATA nor SEEK_HOLE. Define as -1
    so that lseek will return -1 (with errno set to EINVAL).
3.  gcc correctly warns that error is assigned but not used in
    image_copyout_region().  Fix by returning on the first error.
2016-09-26 04:14:00 +00:00
marcel
741bf1228e Avoid depending on the <sys/endian.h> header for le*enc and be*enc.
Not only is the header unportable, the encoding/decoding functions
are as well.  Instead, duplicate the handful of small inlines we
need into a private header called endian.h.

Aside: an alternative approach is to move the encoding/decoding
functions to a separate system header.  While the header is still
nonportable, such an approach would make it possible to re-use the
definitions by playing games with include paths. This may be the
preferred approach if more (build) utilities need this.  This
change does not preclude that.  In fact, it makes it easier.
2016-09-26 01:06:32 +00:00
marcel
0d01313de9 Eliminate the use of EDOOFUS. The error code was used to signal
programming errors, but is really a poor substitute for assert.
And less portable as well.
2016-09-26 00:41:08 +00:00
marcel
a70a0571cb Replace the use of linker sets with constructors for both the
formats and schemes.  Formats and schemes are registered at
runtime now, rather than collected at link time.
2016-09-25 22:57:59 +00:00
marcel
89337d7b72 Update local variable 'block' after calling capacity_resize(),
otherwise format_resize(), which is called right after, isn't
getting the current/actual image size. Rather than rounding up,
format_resize() could end up truncating the size and we don't
allow that by design.

MFC after:	1 week
2016-09-24 17:29:27 +00:00
asomers
a99aa46ed5 Update mkimg(1) author's contact info
Reviewed by:	marcel, imp
MFC after:	3 days
Sponsored by:	Spectra Logic Corp
Differential Revision:	https://reviews.freebsd.org/D7991
2016-09-21 18:42:17 +00:00
pfg
80204cc623 mkimg(1): minor cleanups with argument order in calloc(3).
Generally the first argument in calloc is supposed to stand for a count
and the second for a size. Try to make that consistent. While here,
attempt to make some use of the overflow detection capability in
calloc(3).
2016-07-12 15:46:53 +00:00
emaste
6e628f4e27 mkimg: bump version to 20151211 after r292082
mkimg has had a number of functional additions after the last time the
version was incremented. Do so now, to r292082's commit date, so that
users can determine what is supported.

Reviewed by:	marcel
Approved by:	re (gjb)
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6882
2016-06-17 18:49:42 +00:00
markj
be9a20239f mkimg: Indicate that input file pages are unlikely to be reused.
mkimg(1) uses a swap file to back input file chunks. When the output file
is being written out, blocks of the swap file are mapped and their contents
copied. This causes the backing VM pages to enter the active queue, and when
the output file is large relative to system memory (as is generally the
case), can result in a shortfall of inactive memory. This causes the
pagedaemon to aggressively scan the active queue and swap out process
memory in an attempt to meet the shortfall. Because mkimg's input files
are typically the intermediate result of some build process, there's no
need to push them all through the active queue. Use madvise(2) to indicate
that the backing pages may be reclaimed in preference to active pages. In
the case of the swap file, these pages will be freed as soon as mkimg
exits anyway.

When using mkimg on a desktop-class system with large amounts of dirty
process memory, this change substantially improves mkimg runtime and
reduces swap usage.

Reviewed by:	marcel
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D6654
2016-06-01 02:30:06 +00:00
ngie
92100036c8 Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed
after r298107

Summary of changes:

- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
  namespacing is kept with FILES appropriately, and that this shouldn't need
  to be repeated if the namespace changes -- only the definition of PACKAGE
  needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
  `tests`. In the event we get to the point where things can be split up
  enough in the base system, it would make more sense to group the tests
  with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
  previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
  bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
  ${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
  and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)

Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.

MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division
2016-05-04 23:20:53 +00:00
gjb
1c7e318a9a MFH
Sponsored by:	The FreeBSD Foundation
2016-03-10 21:16:01 +00:00
bdrewery
aab40fdc3d DIRDEPS_BUILD: Connect MK_TESTS.
Sponsored by:	EMC / Isilon Storage Division
2016-03-09 22:46:01 +00:00
gjb
a6998ad84f First pass to fix the 'tests' packages.
Sponsored by:	The FreeBSD Foundation
2016-02-02 22:26:49 +00:00
imp
8bd017c5ae Add ppcboot FAT type. Needed to create a bootable powerpc image.
Differential Review: https://reviews.freebsd.org/D4407
2015-12-11 05:39:42 +00:00
bdrewery
fe1ef27d4a META MODE: Update dependencies with 'the-lot' and add missing directories.
This is not properly respecting WITHOUT or ARCH dependencies in target/.
Doing so requires a massive effort to rework targets/ to do so.  A
better approach will be to either include the SUBDIR Makefiles directly
and map to DIRDEPS or just dynamically lookup the SUBDIR.  These lose
the benefit of having a userland/lib, userland/libexec, etc, though and
results in a massive package.  The current implementation of targets/ is
very unmaintainable.

Currently rescue/rescue and sys/modules are still not connected.

Sponsored by:	EMC / Isilon Storage Division
2015-12-01 05:23:19 +00:00
eadler
21a3003f8f Fix a ton of speelling errors
arc lint is helpful

Reviewed By: allanjude, wblock, #manpages, chris@bsdjunk.com
Differential Revision: https://reviews.freebsd.org/D3337
2015-10-21 05:37:09 +00:00
emaste
c4e448c582 mkimg: support fat16b partitions (MBR type 06h)
Reviewed by:	marcel
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3894
2015-10-15 01:09:14 +00:00
ngie
dd1f618367 Refactor the test/ Makefiles after recent changes to bsd.test.mk (r289158) and
netbsd-tests.test.mk (r289151)

- Eliminate explicit OBJTOP/SRCTOP setting
- Convert all ad hoc NetBSD test integration over to netbsd-tests.test.mk
- Remove unnecessary TESTSDIR setting
- Use SRCTOP where possible for clarity

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Divison
2015-10-12 08:16:03 +00:00
brueffer
2dc82f9774 Fix a few mandoc warnings.
MFC after:	1 week
2015-08-12 10:34:05 +00:00
marcel
37bc3330dc Fix typo introduced in previous commit.
Pointed out by: Nikolai Lifanov <lifanov at mail.lifanov.com>
2015-08-07 18:40:44 +00:00