Commit Graph

178 Commits

Author SHA1 Message Date
Emmanuel Vadot
ea07005137 release: Add an image for CI
A lot of projects CI can't do FreeBSD tests currently.
The main reason is that the project CI infrastructure is runned on Linux
and that our images aren't modifiable from a Linux hosts.
Add a basic image specific for this case (called BASIC-CI for a lack of a
better name).
The image have no package pre-installed.
It only have a few modification to have dhcp client runned on the default
interface and sshd started with option to be able to log on without a password
as root.

Sponsored by: The FreeBSD Foundation

Reviewed by:	re (gjb@)
Differential Revision:	https://reviews.freebsd.org/D25598
2020-11-02 21:10:49 +00:00
Mitchell Horne
1d76e25a67 arm64: convert virtual machine images to GPT
These images were switched to MBR in r281876 as a way to cope with a
hard-coded partition GUID in QEMU's default EFI firmware. Enough time
has passed that this is no longer a problem; QEMU versions >= 4.0
include a copy of edk2 EFI firmware that can detect the root filesystem
properly. Alternatively, sysutils/u-boot-qemu-arm64 can be used.

Switch back to building these images with a GPT partition table, and
re-enable the swap partition.

Reviewed by:	gjb, emaste
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D26986
2020-10-30 18:20:52 +00:00
Mitchell Horne
0d51f9e589 vmimage.subr: noisier failure for unsupported targets
The return code of write_partition_layout() doesn't bubble up, so an
invocation of make vm-release for an incorrect/unsupported target will
appear to succeed while make vm-install will fail due to missing
files. This isn't a common point of failure, but is worth handling
properly.

Upgrade this case to print a message to stderr, and exit in place. This
is okay to do since at this point in the execution of mk-vmimage.sh,
cleanup() has already been run.

Reviewed by:	gjb
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D26985
2020-10-30 18:18:25 +00:00
Mitchell Horne
027b4efccf Slight refactor in vmimage.subr
De-duplicate the invocation of mkimg(1). No functional change.

Reviewed by:	gjb
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D26984
2020-10-30 18:16:10 +00:00
Colin Percival
a72eeb2025 Spawn the DHCPv6 client in EC2 instances via rtsold.
Prior to this commit, EC2 AMIs used a "dual-dhclient" tool which was
launched in place of dhclient and spawned both the base system dhclient
for IPv4 and the ISC dhclient from ports for IPv6.

Now that rtsold supports the "M bit" (managed configuration), we can go
back to having the base system dhclient spawned normally, and provide a
script to rtsold which spawns the ISC dhclient from ports when rtsold
decides that it is appropriate.

Thanks to:	bz
MFC after:	1 week
Sponsored by:	https://www.patreon.com/cperciva
2020-09-13 19:56:53 +00:00
Colin Percival
ff5188db54 Bump the size of EC2 AMIs up to 5 GB.
The FreeBSD base system continues to expand.  4GB is now insufficient;
we passed 3 GB in May 2019; we passed 2 GB in August 2017.  Over half
of the disk space used is in /usr/lib/debug/.

Without this change, instances boot but are unusable, since the first
thing which breaks when VM filesystems are too small is the "pkg install"
in the VM building process.
2020-09-13 19:11:45 +00:00
Oleksandr Tymoshenko
083c262899 Enable EFI system partition on amd64 and i386 VM images
EFI support is a hard requirement for generating Hyper-V Gen2 VM images.

Reviewed by:	gjb
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D25655
2020-07-14 18:02:24 +00:00
Colin Percival
fd5f7f341b Make EC2 AMIs use portsnap and freebsd-update mirrors hosted in AWS
This adjusts freebsd-update.conf and portsnap.conf files in EC2 AMIs to
point at the new AWS-hosted mirror network.

Approved by:	re (delphij)
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D25498
2020-06-30 06:14:34 +00:00
Emmanuel Vadot
45d0192ef8 release: Fix arm GPT image
msdosfs labels are capitalized, use EFI instead of efi.

MFC after:	3 days
2020-06-10 14:10:48 +00:00
Glen Barber
537ab18306 Include the shells/bash port on Vagrant images, which prevents
a shell issue during startup.

PR:		245051
MFC after:	1 minute (if approved by re@)
X-MFC-for:	11.4-RC2
Sponsored by:	Rubicon Communications, LLC (netgate.com)
2020-05-28 18:48:30 +00:00
Colin Percival
5f148eeefd Add ebsnvme-id to EC2 AMIs and enable /dev/aws/disk
The ebsnvme-id utility exposes information about EC2 disks -- for
Elastic Block Store volumes, their volume IDs and "linux device
names", and for Instance Store (aka "Ephemeral") disks, their
serial numbers.

The dev_aws_disk rc.d script and associated devd.conf rule maintains
a tree under /dev/aws/disk:
	/dev/aws/disk/ebs/<volume ID>
	/dev/aws/disk/linuxname/<linux device name>
	/dev/aws/disk/ephemeral/<serial number>
which are symlinks to the corresponding nda or nvd devices.

MFC after:	1 week
Sponsored by:	https://www.patreon.com/cperciva
2020-05-18 02:14:25 +00:00
Colin Percival
ba97bfb0de Add /etc/autofs/special_efs to EC2 AMIs
Since Amazon Elastic File System is only available within AWS, it seems
more appropriate to have this added only in EC2 AMIs rather than
"polluting" non-EC2 images with it.

Reviewed by:	gjb
MFC after:	7 days
Relnotes:	Amazon EFS filesystems can be automounted by enabling autofs
		and placing "/efs -efs" into /etc/auto_master.
Sponsored by:	https://www.patreon.com/cperciva
Differential Revision:	https://reviews.freebsd.org/D24791
2020-05-17 21:54:59 +00:00
Colin Percival
b9f767ebd2 Set use_nvd=0 in EC2 AMIs.
FreeBSD is in the process of switching from nvd(4) to nda(4) as the disk
device front-end to NVMe. Changing the default in the kernel is tricky
since existing systems may have /dev/nvd* hard-coded e.g. in /etc/fstab;
however, there's no reason to not change the default in HEAD for *new*
systems.

At present I have no intention of MFCing this to stable branches, since
someone might reasonably expect scripts they use for launching and
configuring FreeBSD 12.1 instances to work with FreeBSD 12.2 AMIs, for
example.

Reviewed by:	gjb, imp
Relnotes:	NVMe disks in EC2 instances launched from 13.0 and later
		now show up as nda(4) devices.
Differential Revision:	https://reviews.freebsd.org/D24583
2020-04-27 21:44:02 +00:00
Emmanuel Vadot
38e4df107d Add support for generating release images using GPT for ARM
Submitted by:	Daniel Engberg (Original version)
Differential Revision:	https://reviews.freebsd.org/D22537
2020-04-24 16:31:27 +00:00
Colin Percival
9f7cca9218 Mount /dev while deinstalling pkg inside EC2 AMIs.
This gets rid of (harmless) warnings:
"pkg: Cannot open /dev/null:No such file or directory".

MFC after:	2 weeks
2020-02-11 04:05:45 +00:00
Colin Percival
d82420cbfc Remove /qemu from EC2 ARM AMIs
I forgot to do this as part of r345858 -- I added it to the
vm_extra_pre_umount in vmimage.subr but forgot that function
was overridden in the EC2 build.

MFC after:	2 weeks
2020-02-11 04:03:22 +00:00
Warner Losh
2e08f5b4ce multi-boot for openstack/qcow images
Make stock FreeBSD more useful for people wishing to use them. The
QEMU folks suggested this change. It adds a serial console which
allows them to interact with FreeBSD from the earliest moments. This
allows them to configure FreeBSD via the serial port to set it up for
CI use.

Reviewed by: kevans@
Sponsored by: Netflix, Inc
Differential Revision: https://reviews.freebsd.org/D22786
2020-01-27 22:40:03 +00:00
Colin Percival
4faa50e5c2 Switch EC2 AMIs from using the dual-dhclient script to using the new
dual-dhclient-daemon daemon.  This makes it possible to stop/restart
the dhclients.

MFC after:	1 month
2019-10-02 21:35:39 +00:00
Glen Barber
c1f6499260 Fix passing ${CONF_FILES} (which contains MAKE_CONF and
SRC_CONF, __MAKE_CONF and SRCCONF, respectively) through
to arm_install_base() and chroot_arm_build_release().
This prevents failures when the target image is intended
to be build with make.conf(5) and src.conf(5) overrides,
which are correctly handled for non-embedded image builds.

Reported and tested by:	Daniel Engberg
PR:		238615
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2019-06-17 22:53:39 +00:00
Glen Barber
899e81f9f8 Revert r348438.
The fix to override the default python version when building
the sysutils/py-google-compute-engine did not work, and there
are still issues that need to be addressed in the port itself.

See bugzilla 238267 for additional details.

MFC after:	6 days
MFC with:	r348438
MFC note:	no-op to appease the merge tracker
Sponsored by:	The FreeBSD Foundation
2019-05-31 18:40:19 +00:00
Glen Barber
885f5b4a47 Increase the size of the Vagrant default image size, as the 3GB
size is too small to bootstrap the firstboot_pkgs list.

While here, add the growfs(8) startup script to /etc/rc.conf,
as Vagrant images can be resized by modifying the Vagrantfile.

Reported by:	dbaio
PR:		238226
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2019-05-30 17:00:57 +00:00
Glen Barber
5f0c63b621 Fix GCE virtual machine startup.
The ports/head branch recently switched to python3 as the default,
which breaks the sysutils/py-google-compute-engine startup scripts,
as lang/python installs lang/python3{,.x} where lang/python2{,.x}
are needed.

Set DEFAULT_VERSIONS in release/tools/gce.conf to python=2.7, and
remove the lang/python3 inclusion in VM_EXTRA_PACKAGES.

Additionally, unset DEFAULT_VERSIONS in release/tools/vmimage.subr
to prevent persistence of DEFAULT_VERSIONS=python=2.7 in subsequent
VM/cloud image builds.

Note: at present, this affects only 13-CURRENT and 12-STABLE, as
the stable/11 branch had already switched to using the 2019Q2 branch
at the start of the 11.3-RELEASE cycle, so this does not immediately
affect 11.3-BETA, hence the 1-week merge timeout.  This had been
manually tested on 13-CURRENT.

Reported by:	ler (privately)
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2019-05-30 16:49:40 +00:00
Colin Percival
19a60fdc19 Bump EC2 AMI filesystem size up to 4000 MB.
AMIs have been breaking for the past month due to insufficient disk space.

Due to the small amount of overhead in the disk image, the EC2 AMIs end
up with the same (4GB) minimum disk size.

Reported by:	Michal Krawczyk
2019-05-08 21:03:03 +00:00
Glen Barber
a7976e867f Increase the default size of the GCE disk image from 3GB to 20GB,
as 3GB is too small as discovered in this week's snapshot builds.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2019-05-03 17:23:08 +00:00
Glen Barber
cbc5290710 Reduce the default image size for virtual machine disk images from
30GB to 3GB.  The raw images can be resized using truncate(1), and
other formats can be resized with tools included with other tools
included with other hypervisors.

Enable the growfs(8) rc(8) at firstboot if the disk was resized
prior to booting the virtual machine for the first time.

Discussed with:	several
PR:		232313 (requested in other context)
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2019-04-30 14:29:09 +00:00
Colin Percival
1ecaa3f75e Add support for cross-building cloudware images.
If MACHINE_ARCH doesn't match TARGET_ARCH, and we're not in the special
case of building i386 images on an amd64 host, we need to pull in the
qemu-user-static package; this allows us to run some commands inside
the VM disk image chroot, most notably to install packages.

Reviewed by:	gjb
MFC after:	2 weeks
Sponsored by:	FreeBSD/EC2 patreon (https://www.patreon.com/cperciva)
2019-04-03 21:54:47 +00:00
Colin Percival
b55bb76366 Only install amazon-ssm-agent into amd64 AMIs.
This package does not exist on aarch64 at present.
2019-03-20 07:24:21 +00:00
Colin Percival
827e6ec07d Fix sed script to insert Amazon NTP server into ntp.conf once rather
than twice.

Reported by:	Rafal Lukawiecki
MFC after:	1 week
2019-02-19 23:24:39 +00:00
Colin Percival
8cb623ce6e Turn off ec2_ephemeralswap for now
This script broke around FreeBSD 11.0 as a result of SWAPMETA no longer
being reported by vmstat -z; but it also needs to be reworked due to the
arrival in EC2 of nvme ephemeral disks.

I'll turn this option back on after I've found time to rewrite the
script in question.

PR:		234686
Reported by:	meta@
MFC after:	1 week
2019-01-09 03:55:25 +00:00
Rebecca Cran
db8b561345 Rework UEFI ESP generation
Currently, the installer uses pre-created 800KB FAT12 filesystems that
it dd's onto the ESP partition.
This changeset improves that by having the installer generate a FAT32
filesystem directly onto the ESP using newfs_msdos and then copying
loader.efi into /EFI/freebsd.
For live installs it then runs efibootmgr to add a FreeBSD boot entry
in the BIOS.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D17947
2018-12-20 19:39:37 +00:00
Glen Barber
6b26cacc52 Fix NTP query on GCE due to unresolved hostname.
PR:		232456
Submitted by:	Lucas Kanashiro
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-11-26 17:00:39 +00:00
Emmanuel Vadot
9848333b6c release: arm64: Add PINEBOOK config
Add a configuration for PINEBOOK image.
Pinebook is a arm64 laptop based on a Pine64 board.

Since the usb trackpad need a quirk, add a common function for adding
quirk for arm board.
A default one is supplied as most board to not need quirks.

Reviewed by:	gjb
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D18337
2018-11-26 16:38:39 +00:00
Glen Barber
d4e77c31a2 Reduce the GCE image size to 27G to be lower than the free
quota limit.

PR:		232313
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-10-24 15:51:55 +00:00
Kyle Evans
9de0425ed9 release.sh: disable colors and the beastie menu for ARM/ARM64 targets
lualoader has moved to a model where the user is expected to disable color
as desired, rather than disabling it automatically for serial boots, due to
more wide-spread support for color sequences.

In a similar vain, though also to reduce special cases, lualoader no
longer disables the beastie menu automatically for !x86. This was done in
Forth land with a different loader.rc that simply didn't invoke the menu
routines, thus wasn't necessary.

This set of changes puts release images back to how they would've been
experienced prior to the switch to Lua.

Approved by:	re (rgrimes)
2018-08-30 18:00:28 +00:00
Colin Percival
a1b042f79d Disable atkbd0 and atkdbc0 in EC2 AMIs. This has the effect of skipping
the probing and attaching of the PS/2 mouse (not present on EC2) and
keyboard (emulated, but not accessible via EC2).

Note that we disable atkbd0 separately even though during device probing
it shows up as a child of atkbdc0; this is necessary because the device
is also initialized during the early console setup from hammer_time.

This change cuts the kernel boot time on an EC2 c5.4xlarge instance from
7259ms down to 4727 ms.

Approved by:	re (marius)
2018-08-26 03:56:54 +00:00
Emmanuel Vadot
66e380123e release: arm: Setup overlays if board config defines some
Approved by:	re (gjb)
2018-08-24 15:01:22 +00:00
Warner Losh
3e6e16b4da Copy the boot loader from the new location for the co-existing
loaders.

Reviewed by: gjb@
2018-08-17 20:41:50 +00:00
Glen Barber
024b984d17 Add a space between a variable and escaped new line.
MFC after:	3 days
MFC with:	r337717
Sponsored by:	The FreeBSD Foundation
2018-08-13 17:24:31 +00:00
Glen Barber
f75f16f8a7 Add lang/python2, lang/python3, and lang/python to GCE images
to help avoid hard-coding 'python<MAJOR>.<MINOR>' in several
scripts in the client-side scripts.

PR:		230248
MFC after:	3 days
Submitted by:	gustavo.scalet@collabora.com
Sponsored by:	The FreeBSD Foundation
2018-08-13 17:23:43 +00:00
Glen Barber
6ac4304a0f Invoke the growfs rc script for each boot on GCE.
PR:		230275
Submitted by:	gustavo.scalet@collabora.com
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-08-09 23:43:10 +00:00
Glen Barber
0366e18e5a Update and replace old rc daemons for GCE images.
PR:		229000
Submitted by:	helen.koike@collabora.com
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-08-09 23:31:18 +00:00
Emmanuel Vadot
7b7dbfa9b3 release: arm: Copy the dtb to the fat partition
When booting via EFI on arm we have no way to know the dtb file to load
and we always use the one provided from the bootloader.
This works in most case but :

 U-Boot have some really old DTB for some boards, the sync from Linux isn't done automatically for all boards
 Some boards (like TI BeagleBone series) use one u-boot for all the model and it doesn't embed the DTBs
 Some boards (like IMX6 based ones), don't embed the DTB

We want u-boot to load and patch the DTB with the mac address or the display
node enabled or not.

Reviewed by:	gjb, imp
Differential Revision:	https://reviews.freebsd.org/D16596
2018-08-06 17:21:20 +00:00
Emmanuel Vadot
a8d0ee755c release: arm: Enable multicons for arm64
Since we have now EFI framebuffer enabled for ARM64 if we boot on a board
with an screen, u-boot will set up a EFI GOP framebuffer and we won't boot
using the serial console.
Also on RPI3 the firmware always setup the framebuffer area resulting in u-boot
always setup the EFI GOP and FreeBSD never using the serial console.

Reviewed by:	gjb, lwshu (previous version)
Differential Revision:	https://reviews.freebsd.org/D16472
2018-07-31 19:13:50 +00:00
Conrad Meyer
858178a142 Remove insecure ciphers from GCE sshd configuration
They were added for unclear reasons in r277263.  The current OpenSSH
defaults (7.5+) are reasonable, and do not include the insecure rc4 cipher:

                   chacha20-poly1305@openssh.com,
                   aes128-ctr,aes192-ctr,aes256-ctr,
                   aes128-gcm@openssh.com,aes256-gcm@openssh.com,
                   aes128-cbc,aes192-cbc,aes256-cbc

I think I recall there being a reason for a specific list of ciphers on GCE
at the time, but I do not recall what it was, and cannot find any
current GCE documentation of such a list.

So, just revert the explicit configuration and use sane openssh defaults.

PR:		230092
Submitted by:	Gustavo Scalet <gustavo.scalet AT collabora.com>
MFC after:	3 days
Security:	yes
2018-07-28 19:35:49 +00:00
Emmanuel Vadot
43e289df5a release: Add arm_install_boot to install the commit boot bits
This reduce the per-board arm_install_uboot to just install u-boot.
While here remove the installation of rpi.dtb and rpi2.dtb as we load
them from the UFS partition via ubldr.

Reviewed by:	gjb, imp (older version)
MFC after:	3 days
Differential Revision:	https://reviews.freebsd.org/D16239
2018-07-22 12:03:17 +00:00
Edward Tomasz Napierala
4fe984fdae Enable USB OTG serial terminal on ARM SD card images. This configures
the system to make use of USB device mode / USB OTG to provide a "virtual
serial port" on release images.

Reviewed by:	gjb@
MFC after:	2 weeks
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D15602
2018-06-12 16:45:52 +00:00
Edward Tomasz Napierala
a4ccdf9091 Revert r333493, which was a temporary fix for 11.2-RELEASE, and instead
switch the default kldxref_enable to YES.

The reason is that it's required for every image that's being cross-built,
as kldxref(8) cannot handle files for non-native architectures.  For the
one that is not - amd64 - having it on by default doesn't change anything;
the script is noop if the linker.hints already exists.

MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2018-05-26 11:13:17 +00:00
Edward Tomasz Napierala
896eb70c80 Set kldxref_enable="YES" for ARM images. Without it, the images are missing
the /boot/kernel/linker.hints file, which breaks loading some of the modules
with dependencies, eg cfiscsi.ko.

This is a minimal fix for ARM images, in order to safely MFC it before
11.2-RELEASE.  Afterwards, however, I believe we should actually just change
the default (as in, etc/defaults/rc.conf).  The reason is that it's required
for every image that's being cross-built, as kldxref(1) cannot handle files
for non-native architectures.  For the one that is not - amd64 - having it
on by default doesn't change anything - the script is noop if the linker.hints
already exists.

The long-term solution would be to rewrite kldxref(1) to handle other
architectures, and generate linker.hints at build time.

Reviewed by:	gjb@
MFC after:	3 days
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D14534
2018-05-11 14:52:35 +00:00
Glen Barber
ef470d031a Fix a typo.
Submitted by:	lidl
MFC after:	3 days
MFC with:	r333262
Sponsored by:	The FreeBSD Foundation
2018-05-04 21:17:29 +00:00
Glen Barber
d55f0e89c8 Ensure the ports and src trees are available on GCE images,
satisfying a requirement to allow FreeBSD to be considered
a top-tier supported OS in Google Compute Engine.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-05-04 20:38:26 +00:00