Commit Graph

126 Commits

Author SHA1 Message Date
Emmanuel Vadot
8f3c71c85e mkimg: Add support for offset if the source is an image
This allow us to create image with the following format:
mkimg -v -o sdcard -s gpt -p efi:=esp_aarch64.img:1M -p freebsd-ufs::1G

Which will add a efi partition at a 1M offset on the image with its content
coming from the esp_aarch64.img file.

MFC after:	3 days
2021-02-25 18:11:50 +01:00
Emmanuel Vadot
3b6268bb9e mkimg: We always want the last block of the last inserted partition
Even with an absolute offset we want to know the last block the partition
otherwise we endup with an image the size of the metadata.
This allow to create image with the ESP placed at a specific position which
is useful on arm/arm64 where u-boot have always a hard time to read the ESP
if it's not aligned on 512k.
mkimg -v -o sdcard -s gpt -p efi::54M:1M -p freebsd-ufs::1G
now works.

MFC after:	3 days
2021-02-25 16:34:28 +01:00
Nathan Whitehorn
f7aebfbe5c Add GPT PREP-boot type to mkimg(1) from geom_gpt.
This partition type can be used to boot some PowerKVM VMs. We don't
support it well because of some limitations in SLOF, but it's worth at
least have feature parity in geom and mkimg.
2021-02-24 10:31:44 -05:00
Alex Richardson
fe41c64b57 Fix -Wpointer-sign warnings in makefs and mkimg
Reviewed By:	emaste
Differential Revision: https://reviews.freebsd.org/D27175
2021-01-07 09:26:21 +00:00
Alex Richardson
3ac62888fc Significantly speed up mkimg_test
It turns out that the majority of the test time for the mkimg tests isn't
mkimg itself but rather the use of jot and hexdump which can be quite slow
on emulated platforms such as QEMU.

On QEMU-RISC-V this reduces the time for `kyua test mkimg_test` from 655
seconds to 200. And for CheriBSD on QEMU-CHERI this saves 4-5 hours (25%
of the time for the entire testsuite!) since jot ends up triggering slow
functions inside the QEMU emulation a lot.

Reviewed By:	lwhsu
Differential Revision: https://reviews.freebsd.org/D26796
2020-10-18 18:35:23 +00:00
Gordon Bergling
3d265fce43 Fix a few mandoc issues
- skipping paragraph macro: Pp after Sh
- sections out of conventional order: Sh EXAMPLES
- whitespace at end of input line
- normalizing date format
2020-10-09 19:12:44 +00:00
Kyle Evans
7cc42f6d25 Do a sweep and remove most WARNS=6 settings
Repeating the default WARNS here makes it slightly more difficult to
experiment with default WARNS changes, e.g. if we did something absolutely
bananas and introduced a WARNS=7 and wanted to try lifting the default to
that.

Drop most of them; there is one in the blake2 kernel module, but I suspect
it should be dropped -- the default WARNS in the rest of the build doesn't
currently apply to kernel modules, and I haven't put too much thought into
whether it makes sense to make it so.
2020-10-01 01:10:51 +00:00
Oleksandr Tymoshenko
e331252340 Fix invalid VHDX generation for image larger than 4Gb
- Part of BAT payload location was lost due to invalid
    BAT entry encoding type (32 bits instead of 64 bits)
- The sequence of PB/SB entries in BAT was broken due to
    off-by-one index check. It worked for smaller than
    4Gb because there were no SB entries in BAT.

MFC after:	1 day
2020-07-13 02:24:31 +00:00
Baptiste Daroussin
5b990a9463 Revert r362466
Such change should not have happen without prior discussion and review.

With hat:	transitioning core
2020-06-22 07:46:24 +00:00
Hans Petter Selasky
7747001b12 Improve wording to be more precise and clear.
No functional change intended.

s/Master Boot/Main Boot/ (also called MBR)

MFC after:	1 week
Sponsored by:	Mellanox Technologies
2020-06-21 13:34:08 +00:00
Mateusz Piotrowski
19870105e1 Use Fl instead of Ar for long flags
Also, bump date after r361935.

MFC after:	1 week
2020-06-08 21:11:34 +00:00
Oleksandr Tymoshenko
c78cd98b8a Add VHDX support to mkimg(1)
VHDX is the successor of Microsoft's VHD file format. It increases
maximum capacity of the virtual drive to 64TB and introduces features
to better handle power/system failures.

VHDX is the required format for 2nd generation Hyper-V VMs.

Reviewed by:	marcel
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D25184
2020-06-08 20:28:32 +00:00
Alex Richardson
a963845304 Allow bootstrapping mkimg on macOS/Linux
On these systems the (u)int64_t typedefs will not be implicitly defined by the
previous includes, so include <stdint.h> in the header that uses uint64_t.

Reviewed By:	brooks
Differential Revision: https://reviews.freebsd.org/D23202
2020-01-16 14:14:55 +00:00
Marcel Moolenaar
370d612f5e Don't compress and uuencode the "hexdump -C" output files. Just
save them with the $FreeBSD$ tag prepended.  Changes to these
files are now a lot easier to comprehend, which makes diffs also
reviewable.
2019-03-09 02:03:07 +00:00
Marcel Moolenaar
835adc6898 Round # partitions up to fill the last GPT table sector
Set the number of partitions entries in the GPT header to a
multiple of the number of entries that fit in a sector.

PR:		236238
Reviewed by:	imp
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D19465
2019-03-05 22:55:33 +00:00
Xin LI
4dbd43cd05 Remove unneeded headers.
MFC after:	1 month
2019-01-06 20:39:23 +00:00
Benno Rice
10e155c08a Fix a conditional that got mucked up.
Sponsored by:	iXsystems, Inc.
2018-04-12 17:16:13 +00:00
Benno Rice
459a61fd2d Add the ability to specify absolute and relative offsets to size partitions.
To create hybrid boot media we want to specify a partition at a known location.
This extends the syntax of size partitions to include an optional offset that
can be absolute or relative. It also introduces validation to make sure that
this hasn't resulted in overlapping partitions. I haven't added this to the
file and process partition specifications yet but the mechanics are designed
such that if someone comes up with a good way of specifying the offset it
will be fairly easy to add in.

Reviewed by:	imp
Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D14916
2018-04-12 15:47:47 +00:00
Bryan Drewery
ea825d0274 DIRDEPS_BUILD: Update dependencies.
Sponsored by:	Dell EMC Isilon
2017-10-31 00:07:04 +00:00
Enji Cooper
d511b20a69 Add HAS_TESTS to all Makefiles that are currently using the
`SUBDIR.${MK_TESTS}+= tests` idiom.

This is a follow up to r321912.
2017-08-02 08:50:42 +00:00
Enji Cooper
4b330699f8 Convert traditional ${MK_TESTS} conditional idiom for including test
directories to SUBDIR.${MK_TESTS} idiom

This is being done to pave the way for future work (and homogenity) in
^/projects/make-check-sandbox .

No functional change intended.

MFC after:	1 weeks
2017-08-02 08:35:51 +00:00
Enji Cooper
c0c5c2f45a Update the usr.bin/mkimg golden test output files after ^/head@r319125
^/head@r319125 changed the location of the backup pmbr, requiring the
output files to be regenerated, since they're binary disk dumps.

The output files were regenerated with "make rebase"--fixed in
^/head@r319294.

MFC with:	r319125, r319294
PR:		219673
Sponsored by:	Dell EMC Isilon
2017-05-31 08:01:12 +00:00
Enji Cooper
301441dcd8 Fix "make rebase" after ^/head@r315776
"make rebase" can be used for rebasing the output files from mkimg
after making a change to mkimg. This will come in handy soon, per
bug 219673.

MFC after:	3 days
Sponsored by:	Dell EMC Isilon
2017-05-31 07:49:49 +00:00
Enji Cooper
605b213ee1 Formalize the dependent/dependency relationship for <foo>.gz.uu vs <foo>
This helps ensure that the output files are regenerated if the input files
change, after the output files have been created.

MFC after:	3 days
Sponsored by:	Dell EMC Isilon
2017-05-31 07:42:14 +00:00
Emmanuel Vadot
6b14aecae7 mkimg: Correct an off by one error in the PMBR size
The PMBR last sector should be number of sector - 1 (As stated in UEFI Spec
2.6 page 118 table 17).
This fixes warning printed by linux tools like parted or fdisk.

Sponsored by:	Gandi.net
2017-05-29 12:51:02 +00:00
Emmanuel Vadot
2dcfca637d mkimg: Add -C argument to specify maximum capacity
Add a -C option to specify a maximum capacity for the final image file.
It is useful to control the size of the generated image for sdcard or
when we will add dynamic size partition.

Add --capacity which is a shorthand to define min and max capacity at
the same time.

Reviewed by:	bapt, marcel, wblock (manpages)
Sponsored by:	Gandi.net
Differential Revision:	https://reviews.freebsd.org/D10509
2017-05-10 09:36:34 +00:00
Enji Cooper
458cbf0ae6 Rename tests from <foo> to <foo>_test to match the FreeBSD test suite
naming scheme

usr.bin/diff/diff_test was renamed to usr.bin/diff/netbsd_diff_test
to avoid collisions with the renamed FreeBSD test.

MFC after:	1 week
Sponsored by:	Dell EMC Isilon
2017-03-23 03:28:24 +00:00
Pedro F. Giffuni
6613eaa42b mkimg(1): let calloc(3) do the multiplication.
Reviewed by:	marcel
MFC after:	1 week
2017-03-13 20:14:07 +00:00
Simon J. Gerraty
baf4abfc39 Allow building mkimg as cross-tool
For linux the mmap offset must also be page aligned, and we
need to disable macros like __FBSDID()

Change the linux osdep_uuidgen() to use more portable gettimeofday().

Reviewed by: marcel
2017-03-03 01:56:55 +00:00
Baptiste Daroussin
0e630e32be Add missing section in manpage reference
Reported by:	make manlint
MFC after:	2 days
2017-02-11 23:41:38 +00:00
Yoshihiro Takahashi
2b375b4edd Remove pc98 support completely.
I thank all developers and contributors for pc98.

Relnotes:	yes
2017-01-28 02:22:15 +00:00
Warner Losh
c30dcf40ba Add a new flag to mkimg (-a num) to specify the active partition for
those partitioning schemes that have this concept. Implement it as an
override for mbr's setting 0x80 in the flags for the first partition
when we have boot code.

Differential Revision: https://reviews.freebsd.org/D4403
2016-10-18 05:43:12 +00:00
Marcel Moolenaar
4299711473 o Provide a private definition for UUIDs (mkimg_uuid_t) because
UUIDs are not portable.
 o  Move mkimg_uuid() to a new file and merge both gpt_uuid_enc()
    and vhd_uuid_enc() into a single mkimg_uuid_enc() that lives
    in the same file.
 o  Move the OS-specific implementation of generating a UUID to
    osdep_uuidgen() and provide the implementations for FreeBSD,
    macOS and Linux.
 o  Expect the partitioning scheme headers to be found by having
    a search to the directory in which the headers live. This
    avoids conflicts on non-FreeBSD machines.
2016-10-18 01:55:07 +00:00
Marcel Moolenaar
8d249f5e2f Switch to using the portable partition scheme headers. 2016-10-16 02:55:52 +00:00
Marcel Moolenaar
4ac1e8d30f Replace OFF_MAX with INT64_MAX. The former is defined on Linux. 2016-10-03 04:00:30 +00:00
Marcel Moolenaar
6b1235713e Prefer <stdint.h> over <sys/types.h>. While here remove redundant
inclusion of <sys/queue.h>.

Move the inclusion of the disk partitioning headers out of order
and inbetween standard headers and local header. They will change
in a subsequent commit.
2016-10-03 02:37:28 +00:00
Marcel Moolenaar
1080fb197b Replace STAILQ with TAILQ. TAILQs are portable enough that they can
be used on both macOS and Linux. STAILQs are not. In particular,
STAILQ_LAST does not next on Linux. Since neither STAILQ_FOREACH_SAFE
nor TAILQ_FOREACH_SAFE exist on Linux, replace its use with a regular
TAILQ_FOREACH. The _SAFE variant was only used for having the next
pointer in a local variable.
2016-10-03 01:46:47 +00:00
Marcel Moolenaar
7b5a53ea4d Portability changes:
1.  macOS nor Linux have MAP_NOCORE nor MAP_NOSYNC. Define as 0.
2.  macOS doesn't have SEEK_DATA nor SEEK_HOLE. Define as -1
    so that lseek will return -1 (with errno set to EINVAL).
3.  gcc correctly warns that error is assigned but not used in
    image_copyout_region().  Fix by returning on the first error.
2016-09-26 04:14:00 +00:00
Marcel Moolenaar
5aad7d9a87 Avoid depending on the <sys/endian.h> header for le*enc and be*enc.
Not only is the header unportable, the encoding/decoding functions
are as well.  Instead, duplicate the handful of small inlines we
need into a private header called endian.h.

Aside: an alternative approach is to move the encoding/decoding
functions to a separate system header.  While the header is still
nonportable, such an approach would make it possible to re-use the
definitions by playing games with include paths. This may be the
preferred approach if more (build) utilities need this.  This
change does not preclude that.  In fact, it makes it easier.
2016-09-26 01:06:32 +00:00
Marcel Moolenaar
4039ea7c9b Eliminate the use of EDOOFUS. The error code was used to signal
programming errors, but is really a poor substitute for assert.
And less portable as well.
2016-09-26 00:41:08 +00:00
Marcel Moolenaar
5a1302ab2e Replace the use of linker sets with constructors for both the
formats and schemes.  Formats and schemes are registered at
runtime now, rather than collected at link time.
2016-09-25 22:57:59 +00:00
Marcel Moolenaar
ac3425511d Update local variable 'block' after calling capacity_resize(),
otherwise format_resize(), which is called right after, isn't
getting the current/actual image size. Rather than rounding up,
format_resize() could end up truncating the size and we don't
allow that by design.

MFC after:	1 week
2016-09-24 17:29:27 +00:00
Alan Somers
005156d7b3 Update mkimg(1) author's contact info
Reviewed by:	marcel, imp
MFC after:	3 days
Sponsored by:	Spectra Logic Corp
Differential Revision:	https://reviews.freebsd.org/D7991
2016-09-21 18:42:17 +00:00
Pedro F. Giffuni
3af7c80523 mkimg(1): minor cleanups with argument order in calloc(3).
Generally the first argument in calloc is supposed to stand for a count
and the second for a size. Try to make that consistent. While here,
attempt to make some use of the overflow detection capability in
calloc(3).
2016-07-12 15:46:53 +00:00
Ed Maste
023075def4 mkimg: bump version to 20151211 after r292082
mkimg has had a number of functional additions after the last time the
version was incremented. Do so now, to r292082's commit date, so that
users can determine what is supported.

Reviewed by:	marcel
Approved by:	re (gjb)
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6882
2016-06-17 18:49:42 +00:00
Mark Johnston
cc7f01a5da mkimg: Indicate that input file pages are unlikely to be reused.
mkimg(1) uses a swap file to back input file chunks. When the output file
is being written out, blocks of the swap file are mapped and their contents
copied. This causes the backing VM pages to enter the active queue, and when
the output file is large relative to system memory (as is generally the
case), can result in a shortfall of inactive memory. This causes the
pagedaemon to aggressively scan the active queue and swap out process
memory in an attempt to meet the shortfall. Because mkimg's input files
are typically the intermediate result of some build process, there's no
need to push them all through the active queue. Use madvise(2) to indicate
that the backing pages may be reclaimed in preference to active pages. In
the case of the swap file, these pages will be freed as soon as mkimg
exits anyway.

When using mkimg on a desktop-class system with large amounts of dirty
process memory, this change substantially improves mkimg runtime and
reduces swap usage.

Reviewed by:	marcel
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D6654
2016-06-01 02:30:06 +00:00
Enji Cooper
430f7286a5 Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed
after r298107

Summary of changes:

- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
  namespacing is kept with FILES appropriately, and that this shouldn't need
  to be repeated if the namespace changes -- only the definition of PACKAGE
  needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
  `tests`. In the event we get to the point where things can be split up
  enough in the base system, it would make more sense to group the tests
  with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
  previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
  bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
  ${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
  and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)

Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.

MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division
2016-05-04 23:20:53 +00:00
Glen Barber
7d536dc855 MFH
Sponsored by:	The FreeBSD Foundation
2016-03-10 21:16:01 +00:00
Bryan Drewery
15c433351f DIRDEPS_BUILD: Connect MK_TESTS.
Sponsored by:	EMC / Isilon Storage Division
2016-03-09 22:46:01 +00:00
Glen Barber
43faedc133 First pass to fix the 'tests' packages.
Sponsored by:	The FreeBSD Foundation
2016-02-02 22:26:49 +00:00