Commit Graph

193 Commits

Author SHA1 Message Date
Colin Percival
81075203a0 EC2: Turn off IPv6 DAD
Disable Duplicate Address Detection in EC2 instances.  The networking
configuration in EC2, with IPv6 addresses assigned by DHCPv6 and
host egress filtering, makes "duplicate addresses" impossible.

This speeds up the boot process in EC2 by 2 seconds.

Reviewed by:	kp, imp, bz
MFC after:	2 weeks
Sponsored by:	https://www.patreon.com/cperciva
Differential Revision:	https://reviews.freebsd.org/D33091
2021-11-25 19:08:48 -08:00
Li-Wen Hsu
fbece76095
Update Azure release bits
Imports the changes for building official images on Azure Marketplace,
which fulfill the requirements of Azure and FreeBSD cloud images like
disk layout and UEFI for Gen2 VM, along with some minor improvements like
configurations to speed up booting.

"CLOUDWARE" list will be updated after some more collaborations with re
completed.

Reviewed by:	re (gjb)
Sponsored by:	The FreeBSD Foundation
Technical assistance from:	Microsoft
Differential Revision:	https://reviews.freebsd.org/D23804
2021-10-02 04:59:10 +08:00
Brad Davis
83952a5baa release: allow VM_EXTRA_PACKAGES to be specified in the environment
This is useful for adding extra packages to the build of an AMI.
For example:
	env VM_EXTRA_PACKAGES="zsh" make -C release ec2ami

Approved by:	gjb
MFC after:	1 week
Sponsored by:	Rubicon Communications, LLC ("Netgate")
2021-08-09 10:31:51 -06:00
Brad Davis
be2bc82f18 release: fix copypasta
Approved by:	gjb
MFC after:	1 week
X-MFC-With:	fd17ea8c18
Sponsored by:	Rubicon Communications, LLC ("Netgate")
2021-08-09 10:23:18 -06:00
Brad Davis
fd17ea8c18 release: make pkg installs more robust
Currently pkg(8) will fail to install any package if one is missing, so
make this a loop to prevent one missing package from preventing the rest
from installing.  Seen building an AWS AMI for aarch64 on main and
ebsnvme-id is not available in the repo at the moment.

Approved by:	gjb
MFC after:	1 week
Sponsored by:	Rubicon Communications, LLC ("Netgate")
2021-08-09 09:54:08 -06:00
Jose Luis Duran
3f197b101f release: Remove C-like string comparison operator
Per [(1), i.e. test(1), the string comparison operator should be `=`,
not `==` in sh(1) scripts.

No functional change. FreeBSD's test accepts `==` as an alias for `=`
for compatibility with gnu test's `==` extension.

Reviewed by:	imp@, gjb@
Pull Request:	https://github.com/freebsd/freebsd-src/pull/485
2021-06-30 11:13:51 -06:00
Colin Percival
524260db76 EC2: Tell gptboot to skip its 3 second wait
Nobody is going to hit a key to tell gptboot to select another disk,
so there's no point waiting.
2021-06-21 15:37:15 -07:00
Daniel Engerg
5bffdafd6c Remove tmpfs size and properly format generated fstab for arm
Remove tmpfs size limitation, this breaks make installworld and installation of some packages
Format generated fstab using tabs to make it consistent and readable

MFC after:	1 week
Differential Revision: https://reviews.freebsd.org/D29283
2021-03-17 15:02:05 +01:00
Glen Barber
fa04db4762 release: fix VMSIZE following 1ca8842f3a
truncate(1) is not case-sensitive with regard to setting the size
of a file.  makefs(8), however, does not honor upper-case values.
Update release-specific files and the release(7) manual page to
reflect this.

MFC with:	1ca8842f3a
Submitted by:	ehem_freebsd_m5p.com (original)
Differential Review:	https://reviews.freebsd.org/D28979
Sponsored by:	Rubicon Communications, LLC ("Netgate")
2021-03-04 08:43:02 -05:00
Nathan Whitehorn
2c26d77d98 Remove /boot/efi from mtree, missed in 0b7472b3d8.
This had prevented the bootconfig step from determining if an ESP exists,
resulting in its unconditional setup. On BIOS-booted amd64, this wasn't
harmful, just unnecessary, but it resulted in failed installations on
non-EFI-supporting platforms like powerpc64.

MFC after:	3 days
2021-03-02 11:49:41 -05:00
Nathan Whitehorn
1ca8842f3a Use makefs(8) in release VM-image generation instead of md(4) and newfs.
Using makefs instead reduces the privileges needed to build VM images,
simplifies the script (no need to copy files to a fresh image at the end),
and improves portability by allowing generation of cross-endian images.
As a result of the last, this patch also adds support for generation of
powerpc64 and powerpc64le VM images.

No other changes to the output. Tested and working for both amd64 and
powerpc64 targets.

Reviewed by:	gjb
Differential Revision:	https://reviews.freebsd.org/D28912
2021-02-24 21:16:56 -05:00
Nathan Whitehorn
e70eb40271 Mount the ESP on /boot/efi in VM images.
This follows an earlier change (0b7472b3d8) for mounting the ESP from
systems set up through the installer and should be MFC'ed with it.
2021-02-24 08:18:07 -05:00
Glen Barber
c511a5ab53 release: fix overriding UNAME_r
Sponsored by:	Rubicon Communications, LLC ("Netgate")
2021-02-10 21:23:58 -05:00
Mitchell Horne
d46a2a0013 riscv: allow building virtual machine images
RISC-V has the same booting requirements as arm64 (loader.efi, no legacy
boot options), so generated images for both architectures have the same
partition layout.

Reviewed by:	gjb
Differential Revision:	https://reviews.freebsd.org/D27044
2020-12-08 00:37:11 +00:00
Mitchell Horne
86635d4992 release.sh: add support for RISC-V embedded builds
Since the few existing RISC-V hardware platforms are single board
computers, we can piggyback off of arm/arm64's embedded build support
for generating SD card images.

I don't see a pressing need to change the naming in this file at this
time.

Reviewed by:	gjb, manu
Differential Revision:	https://reviews.freebsd.org/D27043
2020-12-08 00:35:13 +00:00
Emmanuel Vadot
ea07005137 release: Add an image for CI
A lot of projects CI can't do FreeBSD tests currently.
The main reason is that the project CI infrastructure is runned on Linux
and that our images aren't modifiable from a Linux hosts.
Add a basic image specific for this case (called BASIC-CI for a lack of a
better name).
The image have no package pre-installed.
It only have a few modification to have dhcp client runned on the default
interface and sshd started with option to be able to log on without a password
as root.

Sponsored by: The FreeBSD Foundation

Reviewed by:	re (gjb@)
Differential Revision:	https://reviews.freebsd.org/D25598
2020-11-02 21:10:49 +00:00
Mitchell Horne
1d76e25a67 arm64: convert virtual machine images to GPT
These images were switched to MBR in r281876 as a way to cope with a
hard-coded partition GUID in QEMU's default EFI firmware. Enough time
has passed that this is no longer a problem; QEMU versions >= 4.0
include a copy of edk2 EFI firmware that can detect the root filesystem
properly. Alternatively, sysutils/u-boot-qemu-arm64 can be used.

Switch back to building these images with a GPT partition table, and
re-enable the swap partition.

Reviewed by:	gjb, emaste
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D26986
2020-10-30 18:20:52 +00:00
Mitchell Horne
0d51f9e589 vmimage.subr: noisier failure for unsupported targets
The return code of write_partition_layout() doesn't bubble up, so an
invocation of make vm-release for an incorrect/unsupported target will
appear to succeed while make vm-install will fail due to missing
files. This isn't a common point of failure, but is worth handling
properly.

Upgrade this case to print a message to stderr, and exit in place. This
is okay to do since at this point in the execution of mk-vmimage.sh,
cleanup() has already been run.

Reviewed by:	gjb
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D26985
2020-10-30 18:18:25 +00:00
Mitchell Horne
027b4efccf Slight refactor in vmimage.subr
De-duplicate the invocation of mkimg(1). No functional change.

Reviewed by:	gjb
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D26984
2020-10-30 18:16:10 +00:00
Colin Percival
a72eeb2025 Spawn the DHCPv6 client in EC2 instances via rtsold.
Prior to this commit, EC2 AMIs used a "dual-dhclient" tool which was
launched in place of dhclient and spawned both the base system dhclient
for IPv4 and the ISC dhclient from ports for IPv6.

Now that rtsold supports the "M bit" (managed configuration), we can go
back to having the base system dhclient spawned normally, and provide a
script to rtsold which spawns the ISC dhclient from ports when rtsold
decides that it is appropriate.

Thanks to:	bz
MFC after:	1 week
Sponsored by:	https://www.patreon.com/cperciva
2020-09-13 19:56:53 +00:00
Colin Percival
ff5188db54 Bump the size of EC2 AMIs up to 5 GB.
The FreeBSD base system continues to expand.  4GB is now insufficient;
we passed 3 GB in May 2019; we passed 2 GB in August 2017.  Over half
of the disk space used is in /usr/lib/debug/.

Without this change, instances boot but are unusable, since the first
thing which breaks when VM filesystems are too small is the "pkg install"
in the VM building process.
2020-09-13 19:11:45 +00:00
Oleksandr Tymoshenko
083c262899 Enable EFI system partition on amd64 and i386 VM images
EFI support is a hard requirement for generating Hyper-V Gen2 VM images.

Reviewed by:	gjb
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D25655
2020-07-14 18:02:24 +00:00
Colin Percival
fd5f7f341b Make EC2 AMIs use portsnap and freebsd-update mirrors hosted in AWS
This adjusts freebsd-update.conf and portsnap.conf files in EC2 AMIs to
point at the new AWS-hosted mirror network.

Approved by:	re (delphij)
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D25498
2020-06-30 06:14:34 +00:00
Emmanuel Vadot
45d0192ef8 release: Fix arm GPT image
msdosfs labels are capitalized, use EFI instead of efi.

MFC after:	3 days
2020-06-10 14:10:48 +00:00
Glen Barber
537ab18306 Include the shells/bash port on Vagrant images, which prevents
a shell issue during startup.

PR:		245051
MFC after:	1 minute (if approved by re@)
X-MFC-for:	11.4-RC2
Sponsored by:	Rubicon Communications, LLC (netgate.com)
2020-05-28 18:48:30 +00:00
Colin Percival
5f148eeefd Add ebsnvme-id to EC2 AMIs and enable /dev/aws/disk
The ebsnvme-id utility exposes information about EC2 disks -- for
Elastic Block Store volumes, their volume IDs and "linux device
names", and for Instance Store (aka "Ephemeral") disks, their
serial numbers.

The dev_aws_disk rc.d script and associated devd.conf rule maintains
a tree under /dev/aws/disk:
	/dev/aws/disk/ebs/<volume ID>
	/dev/aws/disk/linuxname/<linux device name>
	/dev/aws/disk/ephemeral/<serial number>
which are symlinks to the corresponding nda or nvd devices.

MFC after:	1 week
Sponsored by:	https://www.patreon.com/cperciva
2020-05-18 02:14:25 +00:00
Colin Percival
ba97bfb0de Add /etc/autofs/special_efs to EC2 AMIs
Since Amazon Elastic File System is only available within AWS, it seems
more appropriate to have this added only in EC2 AMIs rather than
"polluting" non-EC2 images with it.

Reviewed by:	gjb
MFC after:	7 days
Relnotes:	Amazon EFS filesystems can be automounted by enabling autofs
		and placing "/efs -efs" into /etc/auto_master.
Sponsored by:	https://www.patreon.com/cperciva
Differential Revision:	https://reviews.freebsd.org/D24791
2020-05-17 21:54:59 +00:00
Colin Percival
b9f767ebd2 Set use_nvd=0 in EC2 AMIs.
FreeBSD is in the process of switching from nvd(4) to nda(4) as the disk
device front-end to NVMe. Changing the default in the kernel is tricky
since existing systems may have /dev/nvd* hard-coded e.g. in /etc/fstab;
however, there's no reason to not change the default in HEAD for *new*
systems.

At present I have no intention of MFCing this to stable branches, since
someone might reasonably expect scripts they use for launching and
configuring FreeBSD 12.1 instances to work with FreeBSD 12.2 AMIs, for
example.

Reviewed by:	gjb, imp
Relnotes:	NVMe disks in EC2 instances launched from 13.0 and later
		now show up as nda(4) devices.
Differential Revision:	https://reviews.freebsd.org/D24583
2020-04-27 21:44:02 +00:00
Emmanuel Vadot
38e4df107d Add support for generating release images using GPT for ARM
Submitted by:	Daniel Engberg (Original version)
Differential Revision:	https://reviews.freebsd.org/D22537
2020-04-24 16:31:27 +00:00
Colin Percival
9f7cca9218 Mount /dev while deinstalling pkg inside EC2 AMIs.
This gets rid of (harmless) warnings:
"pkg: Cannot open /dev/null:No such file or directory".

MFC after:	2 weeks
2020-02-11 04:05:45 +00:00
Colin Percival
d82420cbfc Remove /qemu from EC2 ARM AMIs
I forgot to do this as part of r345858 -- I added it to the
vm_extra_pre_umount in vmimage.subr but forgot that function
was overridden in the EC2 build.

MFC after:	2 weeks
2020-02-11 04:03:22 +00:00
Warner Losh
2e08f5b4ce multi-boot for openstack/qcow images
Make stock FreeBSD more useful for people wishing to use them. The
QEMU folks suggested this change. It adds a serial console which
allows them to interact with FreeBSD from the earliest moments. This
allows them to configure FreeBSD via the serial port to set it up for
CI use.

Reviewed by: kevans@
Sponsored by: Netflix, Inc
Differential Revision: https://reviews.freebsd.org/D22786
2020-01-27 22:40:03 +00:00
Colin Percival
4faa50e5c2 Switch EC2 AMIs from using the dual-dhclient script to using the new
dual-dhclient-daemon daemon.  This makes it possible to stop/restart
the dhclients.

MFC after:	1 month
2019-10-02 21:35:39 +00:00
Glen Barber
c1f6499260 Fix passing ${CONF_FILES} (which contains MAKE_CONF and
SRC_CONF, __MAKE_CONF and SRCCONF, respectively) through
to arm_install_base() and chroot_arm_build_release().
This prevents failures when the target image is intended
to be build with make.conf(5) and src.conf(5) overrides,
which are correctly handled for non-embedded image builds.

Reported and tested by:	Daniel Engberg
PR:		238615
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2019-06-17 22:53:39 +00:00
Glen Barber
899e81f9f8 Revert r348438.
The fix to override the default python version when building
the sysutils/py-google-compute-engine did not work, and there
are still issues that need to be addressed in the port itself.

See bugzilla 238267 for additional details.

MFC after:	6 days
MFC with:	r348438
MFC note:	no-op to appease the merge tracker
Sponsored by:	The FreeBSD Foundation
2019-05-31 18:40:19 +00:00
Glen Barber
885f5b4a47 Increase the size of the Vagrant default image size, as the 3GB
size is too small to bootstrap the firstboot_pkgs list.

While here, add the growfs(8) startup script to /etc/rc.conf,
as Vagrant images can be resized by modifying the Vagrantfile.

Reported by:	dbaio
PR:		238226
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2019-05-30 17:00:57 +00:00
Glen Barber
5f0c63b621 Fix GCE virtual machine startup.
The ports/head branch recently switched to python3 as the default,
which breaks the sysutils/py-google-compute-engine startup scripts,
as lang/python installs lang/python3{,.x} where lang/python2{,.x}
are needed.

Set DEFAULT_VERSIONS in release/tools/gce.conf to python=2.7, and
remove the lang/python3 inclusion in VM_EXTRA_PACKAGES.

Additionally, unset DEFAULT_VERSIONS in release/tools/vmimage.subr
to prevent persistence of DEFAULT_VERSIONS=python=2.7 in subsequent
VM/cloud image builds.

Note: at present, this affects only 13-CURRENT and 12-STABLE, as
the stable/11 branch had already switched to using the 2019Q2 branch
at the start of the 11.3-RELEASE cycle, so this does not immediately
affect 11.3-BETA, hence the 1-week merge timeout.  This had been
manually tested on 13-CURRENT.

Reported by:	ler (privately)
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2019-05-30 16:49:40 +00:00
Colin Percival
19a60fdc19 Bump EC2 AMI filesystem size up to 4000 MB.
AMIs have been breaking for the past month due to insufficient disk space.

Due to the small amount of overhead in the disk image, the EC2 AMIs end
up with the same (4GB) minimum disk size.

Reported by:	Michal Krawczyk
2019-05-08 21:03:03 +00:00
Glen Barber
a7976e867f Increase the default size of the GCE disk image from 3GB to 20GB,
as 3GB is too small as discovered in this week's snapshot builds.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2019-05-03 17:23:08 +00:00
Glen Barber
cbc5290710 Reduce the default image size for virtual machine disk images from
30GB to 3GB.  The raw images can be resized using truncate(1), and
other formats can be resized with tools included with other tools
included with other hypervisors.

Enable the growfs(8) rc(8) at firstboot if the disk was resized
prior to booting the virtual machine for the first time.

Discussed with:	several
PR:		232313 (requested in other context)
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2019-04-30 14:29:09 +00:00
Colin Percival
1ecaa3f75e Add support for cross-building cloudware images.
If MACHINE_ARCH doesn't match TARGET_ARCH, and we're not in the special
case of building i386 images on an amd64 host, we need to pull in the
qemu-user-static package; this allows us to run some commands inside
the VM disk image chroot, most notably to install packages.

Reviewed by:	gjb
MFC after:	2 weeks
Sponsored by:	FreeBSD/EC2 patreon (https://www.patreon.com/cperciva)
2019-04-03 21:54:47 +00:00
Colin Percival
b55bb76366 Only install amazon-ssm-agent into amd64 AMIs.
This package does not exist on aarch64 at present.
2019-03-20 07:24:21 +00:00
Colin Percival
827e6ec07d Fix sed script to insert Amazon NTP server into ntp.conf once rather
than twice.

Reported by:	Rafal Lukawiecki
MFC after:	1 week
2019-02-19 23:24:39 +00:00
Colin Percival
8cb623ce6e Turn off ec2_ephemeralswap for now
This script broke around FreeBSD 11.0 as a result of SWAPMETA no longer
being reported by vmstat -z; but it also needs to be reworked due to the
arrival in EC2 of nvme ephemeral disks.

I'll turn this option back on after I've found time to rewrite the
script in question.

PR:		234686
Reported by:	meta@
MFC after:	1 week
2019-01-09 03:55:25 +00:00
Rebecca Cran
db8b561345 Rework UEFI ESP generation
Currently, the installer uses pre-created 800KB FAT12 filesystems that
it dd's onto the ESP partition.
This changeset improves that by having the installer generate a FAT32
filesystem directly onto the ESP using newfs_msdos and then copying
loader.efi into /EFI/freebsd.
For live installs it then runs efibootmgr to add a FreeBSD boot entry
in the BIOS.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D17947
2018-12-20 19:39:37 +00:00
Glen Barber
6b26cacc52 Fix NTP query on GCE due to unresolved hostname.
PR:		232456
Submitted by:	Lucas Kanashiro
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-11-26 17:00:39 +00:00
Emmanuel Vadot
9848333b6c release: arm64: Add PINEBOOK config
Add a configuration for PINEBOOK image.
Pinebook is a arm64 laptop based on a Pine64 board.

Since the usb trackpad need a quirk, add a common function for adding
quirk for arm board.
A default one is supplied as most board to not need quirks.

Reviewed by:	gjb
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D18337
2018-11-26 16:38:39 +00:00
Glen Barber
d4e77c31a2 Reduce the GCE image size to 27G to be lower than the free
quota limit.

PR:		232313
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-10-24 15:51:55 +00:00
Kyle Evans
9de0425ed9 release.sh: disable colors and the beastie menu for ARM/ARM64 targets
lualoader has moved to a model where the user is expected to disable color
as desired, rather than disabling it automatically for serial boots, due to
more wide-spread support for color sequences.

In a similar vain, though also to reduce special cases, lualoader no
longer disables the beastie menu automatically for !x86. This was done in
Forth land with a different loader.rc that simply didn't invoke the menu
routines, thus wasn't necessary.

This set of changes puts release images back to how they would've been
experienced prior to the switch to Lua.

Approved by:	re (rgrimes)
2018-08-30 18:00:28 +00:00
Colin Percival
a1b042f79d Disable atkbd0 and atkdbc0 in EC2 AMIs. This has the effect of skipping
the probing and attaching of the PS/2 mouse (not present on EC2) and
keyboard (emulated, but not accessible via EC2).

Note that we disable atkbd0 separately even though during device probing
it shows up as a child of atkbdc0; this is necessary because the device
is also initialized during the early console setup from hammer_time.

This change cuts the kernel boot time on an EC2 c5.4xlarge instance from
7259ms down to 4727 ms.

Approved by:	re (marius)
2018-08-26 03:56:54 +00:00