Commit Graph

164 Commits

Author SHA1 Message Date
Colin Percival
9f7cca9218 Mount /dev while deinstalling pkg inside EC2 AMIs.
This gets rid of (harmless) warnings:
"pkg: Cannot open /dev/null:No such file or directory".

MFC after:	2 weeks
2020-02-11 04:05:45 +00:00
Colin Percival
d82420cbfc Remove /qemu from EC2 ARM AMIs
I forgot to do this as part of r345858 -- I added it to the
vm_extra_pre_umount in vmimage.subr but forgot that function
was overridden in the EC2 build.

MFC after:	2 weeks
2020-02-11 04:03:22 +00:00
Warner Losh
2e08f5b4ce multi-boot for openstack/qcow images
Make stock FreeBSD more useful for people wishing to use them. The
QEMU folks suggested this change. It adds a serial console which
allows them to interact with FreeBSD from the earliest moments. This
allows them to configure FreeBSD via the serial port to set it up for
CI use.

Reviewed by: kevans@
Sponsored by: Netflix, Inc
Differential Revision: https://reviews.freebsd.org/D22786
2020-01-27 22:40:03 +00:00
Colin Percival
4faa50e5c2 Switch EC2 AMIs from using the dual-dhclient script to using the new
dual-dhclient-daemon daemon.  This makes it possible to stop/restart
the dhclients.

MFC after:	1 month
2019-10-02 21:35:39 +00:00
Glen Barber
c1f6499260 Fix passing ${CONF_FILES} (which contains MAKE_CONF and
SRC_CONF, __MAKE_CONF and SRCCONF, respectively) through
to arm_install_base() and chroot_arm_build_release().
This prevents failures when the target image is intended
to be build with make.conf(5) and src.conf(5) overrides,
which are correctly handled for non-embedded image builds.

Reported and tested by:	Daniel Engberg
PR:		238615
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2019-06-17 22:53:39 +00:00
Glen Barber
899e81f9f8 Revert r348438.
The fix to override the default python version when building
the sysutils/py-google-compute-engine did not work, and there
are still issues that need to be addressed in the port itself.

See bugzilla 238267 for additional details.

MFC after:	6 days
MFC with:	r348438
MFC note:	no-op to appease the merge tracker
Sponsored by:	The FreeBSD Foundation
2019-05-31 18:40:19 +00:00
Glen Barber
885f5b4a47 Increase the size of the Vagrant default image size, as the 3GB
size is too small to bootstrap the firstboot_pkgs list.

While here, add the growfs(8) startup script to /etc/rc.conf,
as Vagrant images can be resized by modifying the Vagrantfile.

Reported by:	dbaio
PR:		238226
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2019-05-30 17:00:57 +00:00
Glen Barber
5f0c63b621 Fix GCE virtual machine startup.
The ports/head branch recently switched to python3 as the default,
which breaks the sysutils/py-google-compute-engine startup scripts,
as lang/python installs lang/python3{,.x} where lang/python2{,.x}
are needed.

Set DEFAULT_VERSIONS in release/tools/gce.conf to python=2.7, and
remove the lang/python3 inclusion in VM_EXTRA_PACKAGES.

Additionally, unset DEFAULT_VERSIONS in release/tools/vmimage.subr
to prevent persistence of DEFAULT_VERSIONS=python=2.7 in subsequent
VM/cloud image builds.

Note: at present, this affects only 13-CURRENT and 12-STABLE, as
the stable/11 branch had already switched to using the 2019Q2 branch
at the start of the 11.3-RELEASE cycle, so this does not immediately
affect 11.3-BETA, hence the 1-week merge timeout.  This had been
manually tested on 13-CURRENT.

Reported by:	ler (privately)
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2019-05-30 16:49:40 +00:00
Colin Percival
19a60fdc19 Bump EC2 AMI filesystem size up to 4000 MB.
AMIs have been breaking for the past month due to insufficient disk space.

Due to the small amount of overhead in the disk image, the EC2 AMIs end
up with the same (4GB) minimum disk size.

Reported by:	Michal Krawczyk
2019-05-08 21:03:03 +00:00
Glen Barber
a7976e867f Increase the default size of the GCE disk image from 3GB to 20GB,
as 3GB is too small as discovered in this week's snapshot builds.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2019-05-03 17:23:08 +00:00
Glen Barber
cbc5290710 Reduce the default image size for virtual machine disk images from
30GB to 3GB.  The raw images can be resized using truncate(1), and
other formats can be resized with tools included with other tools
included with other hypervisors.

Enable the growfs(8) rc(8) at firstboot if the disk was resized
prior to booting the virtual machine for the first time.

Discussed with:	several
PR:		232313 (requested in other context)
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2019-04-30 14:29:09 +00:00
Colin Percival
1ecaa3f75e Add support for cross-building cloudware images.
If MACHINE_ARCH doesn't match TARGET_ARCH, and we're not in the special
case of building i386 images on an amd64 host, we need to pull in the
qemu-user-static package; this allows us to run some commands inside
the VM disk image chroot, most notably to install packages.

Reviewed by:	gjb
MFC after:	2 weeks
Sponsored by:	FreeBSD/EC2 patreon (https://www.patreon.com/cperciva)
2019-04-03 21:54:47 +00:00
Colin Percival
b55bb76366 Only install amazon-ssm-agent into amd64 AMIs.
This package does not exist on aarch64 at present.
2019-03-20 07:24:21 +00:00
Colin Percival
827e6ec07d Fix sed script to insert Amazon NTP server into ntp.conf once rather
than twice.

Reported by:	Rafal Lukawiecki
MFC after:	1 week
2019-02-19 23:24:39 +00:00
Colin Percival
8cb623ce6e Turn off ec2_ephemeralswap for now
This script broke around FreeBSD 11.0 as a result of SWAPMETA no longer
being reported by vmstat -z; but it also needs to be reworked due to the
arrival in EC2 of nvme ephemeral disks.

I'll turn this option back on after I've found time to rewrite the
script in question.

PR:		234686
Reported by:	meta@
MFC after:	1 week
2019-01-09 03:55:25 +00:00
Rebecca Cran
db8b561345 Rework UEFI ESP generation
Currently, the installer uses pre-created 800KB FAT12 filesystems that
it dd's onto the ESP partition.
This changeset improves that by having the installer generate a FAT32
filesystem directly onto the ESP using newfs_msdos and then copying
loader.efi into /EFI/freebsd.
For live installs it then runs efibootmgr to add a FreeBSD boot entry
in the BIOS.

Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D17947
2018-12-20 19:39:37 +00:00
Glen Barber
6b26cacc52 Fix NTP query on GCE due to unresolved hostname.
PR:		232456
Submitted by:	Lucas Kanashiro
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-11-26 17:00:39 +00:00
Emmanuel Vadot
9848333b6c release: arm64: Add PINEBOOK config
Add a configuration for PINEBOOK image.
Pinebook is a arm64 laptop based on a Pine64 board.

Since the usb trackpad need a quirk, add a common function for adding
quirk for arm board.
A default one is supplied as most board to not need quirks.

Reviewed by:	gjb
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D18337
2018-11-26 16:38:39 +00:00
Glen Barber
d4e77c31a2 Reduce the GCE image size to 27G to be lower than the free
quota limit.

PR:		232313
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-10-24 15:51:55 +00:00
Kyle Evans
9de0425ed9 release.sh: disable colors and the beastie menu for ARM/ARM64 targets
lualoader has moved to a model where the user is expected to disable color
as desired, rather than disabling it automatically for serial boots, due to
more wide-spread support for color sequences.

In a similar vain, though also to reduce special cases, lualoader no
longer disables the beastie menu automatically for !x86. This was done in
Forth land with a different loader.rc that simply didn't invoke the menu
routines, thus wasn't necessary.

This set of changes puts release images back to how they would've been
experienced prior to the switch to Lua.

Approved by:	re (rgrimes)
2018-08-30 18:00:28 +00:00
Colin Percival
a1b042f79d Disable atkbd0 and atkdbc0 in EC2 AMIs. This has the effect of skipping
the probing and attaching of the PS/2 mouse (not present on EC2) and
keyboard (emulated, but not accessible via EC2).

Note that we disable atkbd0 separately even though during device probing
it shows up as a child of atkbdc0; this is necessary because the device
is also initialized during the early console setup from hammer_time.

This change cuts the kernel boot time on an EC2 c5.4xlarge instance from
7259ms down to 4727 ms.

Approved by:	re (marius)
2018-08-26 03:56:54 +00:00
Emmanuel Vadot
66e380123e release: arm: Setup overlays if board config defines some
Approved by:	re (gjb)
2018-08-24 15:01:22 +00:00
Warner Losh
3e6e16b4da Copy the boot loader from the new location for the co-existing
loaders.

Reviewed by: gjb@
2018-08-17 20:41:50 +00:00
Glen Barber
024b984d17 Add a space between a variable and escaped new line.
MFC after:	3 days
MFC with:	r337717
Sponsored by:	The FreeBSD Foundation
2018-08-13 17:24:31 +00:00
Glen Barber
f75f16f8a7 Add lang/python2, lang/python3, and lang/python to GCE images
to help avoid hard-coding 'python<MAJOR>.<MINOR>' in several
scripts in the client-side scripts.

PR:		230248
MFC after:	3 days
Submitted by:	gustavo.scalet@collabora.com
Sponsored by:	The FreeBSD Foundation
2018-08-13 17:23:43 +00:00
Glen Barber
6ac4304a0f Invoke the growfs rc script for each boot on GCE.
PR:		230275
Submitted by:	gustavo.scalet@collabora.com
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-08-09 23:43:10 +00:00
Glen Barber
0366e18e5a Update and replace old rc daemons for GCE images.
PR:		229000
Submitted by:	helen.koike@collabora.com
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-08-09 23:31:18 +00:00
Emmanuel Vadot
7b7dbfa9b3 release: arm: Copy the dtb to the fat partition
When booting via EFI on arm we have no way to know the dtb file to load
and we always use the one provided from the bootloader.
This works in most case but :

 U-Boot have some really old DTB for some boards, the sync from Linux isn't done automatically for all boards
 Some boards (like TI BeagleBone series) use one u-boot for all the model and it doesn't embed the DTBs
 Some boards (like IMX6 based ones), don't embed the DTB

We want u-boot to load and patch the DTB with the mac address or the display
node enabled or not.

Reviewed by:	gjb, imp
Differential Revision:	https://reviews.freebsd.org/D16596
2018-08-06 17:21:20 +00:00
Emmanuel Vadot
a8d0ee755c release: arm: Enable multicons for arm64
Since we have now EFI framebuffer enabled for ARM64 if we boot on a board
with an screen, u-boot will set up a EFI GOP framebuffer and we won't boot
using the serial console.
Also on RPI3 the firmware always setup the framebuffer area resulting in u-boot
always setup the EFI GOP and FreeBSD never using the serial console.

Reviewed by:	gjb, lwshu (previous version)
Differential Revision:	https://reviews.freebsd.org/D16472
2018-07-31 19:13:50 +00:00
Conrad Meyer
858178a142 Remove insecure ciphers from GCE sshd configuration
They were added for unclear reasons in r277263.  The current OpenSSH
defaults (7.5+) are reasonable, and do not include the insecure rc4 cipher:

                   chacha20-poly1305@openssh.com,
                   aes128-ctr,aes192-ctr,aes256-ctr,
                   aes128-gcm@openssh.com,aes256-gcm@openssh.com,
                   aes128-cbc,aes192-cbc,aes256-cbc

I think I recall there being a reason for a specific list of ciphers on GCE
at the time, but I do not recall what it was, and cannot find any
current GCE documentation of such a list.

So, just revert the explicit configuration and use sane openssh defaults.

PR:		230092
Submitted by:	Gustavo Scalet <gustavo.scalet AT collabora.com>
MFC after:	3 days
Security:	yes
2018-07-28 19:35:49 +00:00
Emmanuel Vadot
43e289df5a release: Add arm_install_boot to install the commit boot bits
This reduce the per-board arm_install_uboot to just install u-boot.
While here remove the installation of rpi.dtb and rpi2.dtb as we load
them from the UFS partition via ubldr.

Reviewed by:	gjb, imp (older version)
MFC after:	3 days
Differential Revision:	https://reviews.freebsd.org/D16239
2018-07-22 12:03:17 +00:00
Edward Tomasz Napierala
4fe984fdae Enable USB OTG serial terminal on ARM SD card images. This configures
the system to make use of USB device mode / USB OTG to provide a "virtual
serial port" on release images.

Reviewed by:	gjb@
MFC after:	2 weeks
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D15602
2018-06-12 16:45:52 +00:00
Edward Tomasz Napierala
a4ccdf9091 Revert r333493, which was a temporary fix for 11.2-RELEASE, and instead
switch the default kldxref_enable to YES.

The reason is that it's required for every image that's being cross-built,
as kldxref(8) cannot handle files for non-native architectures.  For the
one that is not - amd64 - having it on by default doesn't change anything;
the script is noop if the linker.hints already exists.

MFC after:	2 weeks
Sponsored by:	DARPA, AFRL
2018-05-26 11:13:17 +00:00
Edward Tomasz Napierala
896eb70c80 Set kldxref_enable="YES" for ARM images. Without it, the images are missing
the /boot/kernel/linker.hints file, which breaks loading some of the modules
with dependencies, eg cfiscsi.ko.

This is a minimal fix for ARM images, in order to safely MFC it before
11.2-RELEASE.  Afterwards, however, I believe we should actually just change
the default (as in, etc/defaults/rc.conf).  The reason is that it's required
for every image that's being cross-built, as kldxref(1) cannot handle files
for non-native architectures.  For the one that is not - amd64 - having it
on by default doesn't change anything - the script is noop if the linker.hints
already exists.

The long-term solution would be to rewrite kldxref(1) to handle other
architectures, and generate linker.hints at build time.

Reviewed by:	gjb@
MFC after:	3 days
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D14534
2018-05-11 14:52:35 +00:00
Glen Barber
ef470d031a Fix a typo.
Submitted by:	lidl
MFC after:	3 days
MFC with:	r333262
Sponsored by:	The FreeBSD Foundation
2018-05-04 21:17:29 +00:00
Glen Barber
d55f0e89c8 Ensure the ports and src trees are available on GCE images,
satisfying a requirement to allow FreeBSD to be considered
a top-tier supported OS in Google Compute Engine.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-05-04 20:38:26 +00:00
Colin Percival
b6028b0ba1 Move debug.{trace,debugger}_on_panic and kern.panic_reboot_wait_time in
EC2 instances from sysctl.conf to loader.conf; these can all be set as
loader tunables, and setting them in loader.conf gives us the right
behaviour in the event of a kernel panic taking place prior to when
sysctl.conf is processed.

MFC after:	1 week
2018-04-18 05:58:27 +00:00
Glen Barber
5db02b50ce Escape trailing newlines in a long variable list for consistency.
Submitted by:	garga
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-03-26 18:24:16 +00:00
Glen Barber
c0505015d6 Remove google_accounts_manager from VM_RC_LIST in the GCE configuration
file, no longer needed.

PR:		221714
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-03-22 17:49:27 +00:00
Colin Percival
4a93691064 Make EC2 instances use Amazon's NTP service for time synchronization.
Since Amazon provides NTP servers within their network, this should
be far superior to using the default NTP pools; and since the service
is provided by Amazon there's very little risk in enabling it by
default.  (If someone is able to compromise Amazon's NTP servers and
exploit them to attack EC2 instances, they would almost certainly be
able to compromise EC2 instances even without ntpd running...)

MFC after:	1 week
Relnotes:	EC2 instances now keep their clocks synchronized using
		the Amazon Time Sync Service (aka. NTP).
2017-12-05 09:22:14 +00:00
Colin Percival
4ba35bc4db Resurrect r321659: Turn off ChallengeResponseAuthentication for EC2 AMIs.
EC2 instances are normally launched with an SSH public key specified,
which is then used for logging in (by default, as 'ec2-user').  Having
ChallengeResponseAuthentication enabled (as FreeBSD's default sshd_config
does) has no functional effect in a new EC2 instance, since you can't log
in using a password until a password has been set -- but having this
enabled results in alerts from automated scanning tools which can detect
that sshd advertises support for keyboard-interactive logins (since they
can't detect that accounts have no password set).

EC2 users who want to use passwords to log in to their instances will need
to set 'ChallengeResponseAuthentication yes' in FreeBSD 12.0 and later.

Discussed with:	gjb, gtetlow, emaste, des
Requested by:	Amazon
X-MFC:		No
Relnotes:	ChallengeResponseAuthentication is turned off by default in
		Amazon EC2 AMIs.
2017-12-05 09:08:48 +00:00
Glen Barber
c205468e00 Fix an indentation nit.
Sponsored by:	The FreeBSD Foundation
2017-11-30 20:52:01 +00:00
Glen Barber
3103eac6a0 Remove /etc/resolv.conf from virtual machine images, which is
copied from the build host.  It is renamed to /etc/resolv.conf.bak
on boot, so never used anyway.

Noticed by:	peter
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-11-21 18:02:18 +00:00
Warner Losh
ca987d4641 Move sys/boot to stand. Fix all references to new location
Sponsored by:	Netflix
2017-11-14 23:02:19 +00:00
Bryan Drewery
dfa099890c Add option UNIFIED_OBJDIR, on by default, which moves the default build OBJDIR.
This changes the build OBJDIR from the older style of /usr/obj/<srcdir> for
native builds, and /usr/obj/<target>.<target_arch>/<srcdir> for cross builds to
a new simpler format of /usr/obj/<srcdir>/<target>.<target_arch>.  This
new format is used regardless of cross or native build.  It allows
easier management of multiple source tree object directories.

The UNIFIED_OBJDIR option will be removed and its feature made permanent
for the 12.0 release.

Relnotes:	yes (don't note UNIFIED_OBJDIR option since it will be removed)
Prior work:	D3711 D874
Reviewed by:	gjb, sjg
Discussed at:	https://lists.freebsd.org/pipermail/freebsd-arch/2016-May/017805.html
Discussed with:	emaste
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D12840
2017-11-01 21:22:05 +00:00
Colin Percival
9d98ae7d5f Add the amazon-ssm-agent package to EC2 AMI builds. This makes it
immediately available on instances which are running without internet
access (or which can't rely on firstboot_pkgs to install it for some
other reason).

Note that this agent is not enabled by default; to enable it, add
amazon_ssm_agent_enable="YES" to /etc/rc.conf, e.g., by placing the lines
	>>/etc/rc.conf
	amazon_ssm_agent_enable="YES"
into the EC2 user-data.  In addition to being enabled, the agent requires
keys to be provided via IAM Roles; users are encouraged to be very careful
in using this functionality due to the inherent vulnerability in the idea
of providing credentials via a service accessible to any process which can
open an HTTP connection.

Requested by:	Amazon
No objection from:	re@
Relnotes:	FreeBSD/EC2 AMIs now include the Amazon EC2 Systems Manager
		(SSM) Agent.
2017-11-01 00:33:54 +00:00
Glen Barber
2170fff8a5 Set a default hostname for virtual machine images.
A recent bug in security/sudo causes segmentation faults when
the system is not configured with a hostname, which causes issues
with some virtual machine setups, notably Vagrant.  Set the default
hostname to the output of 'uname -o'.

Submitted by:	Nicholas Fiorentini
Sponsored by:	The FreeBSD Foundation
2017-10-30 13:54:54 +00:00
Glen Barber
3f04d8c304 Revert r323812 from release/tools/arm.subr, which has broken the
build on arm/armv6 images.

Pointyhat:	gjb (myself)
MFC after:	immediate
MFC note:	releng/10.4 has broken because of this
Sponsored by:	The FreeBSD Foundation
2017-09-22 14:34:27 +00:00
Glen Barber
c52962c68e Bootstrap etcupdate(8) and mergemaster(8) databases when creating
virtual machine images and embedded images, similar to what is
done when extracting base.txz to the target root filesystem in
an new installation.

Noticed by:	marius
Tested with:	head@r323729
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-09-20 15:49:12 +00:00
Glen Barber
505f20a67b Increase the Amazon EC2 AMI image size from 2GB to 3GB to prevent
image build failures due to a full md(4)-backed filesystem.

Sponsored by:	The FreeBSD Foundation
2017-08-28 14:49:26 +00:00