The change extends vmimage.subr to handle a new parameter, VMFS, which
should be equal to either "ufs" or "zfs". When it is set to ZFS, we use
makefs to create a bootable pool populated using the same dataset layout
as bsdinstall and "poudriere image" use. The pool can be grown using
the growfs rc.d script, just as in UFS images.
This will make it easy to provide VM and cloud images with ZFS as the
root filesystem. So far I did not do extensive testing of cloud images;
I merely verified that creation of ZFS-based AWS AMIs works and allows
me to create amd64 and arm64 EC2 instances with ZFS as the root
filesystem.
Reviewed by: emaste, gjb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34426
The latest version of the ec2-scripts package includes a completely
rewritten "use EC2 ephemeral disks for swap space" script. Now that
we have something which works on recent versions of FreeBSD, turn it
on since it's a great way to use the ephemeral disks.
Note that the option for controlling this, ec2_ephemeral_swap_enable,
is not the same as the option (ec2_ephemeralswap_enable) used with the
previous ephemeral-swap script; this change was deliberate to avoid
astonishment for users who upgraded their ec2-scripts package and had
a setting left behind in rc.conf.
This sshd_config keyword was replaced by KbdInteractiveAuthentication in
openssh 8.7, though ChallengeResponseAuthentication is silently accepted
as an alias. However, this means that the code in ec2.conf which
modifies a commented-out line no longer does anything. Apply a minimal
fix.
Reviewed by: cperciva, emaste
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34400
Disable Duplicate Address Detection in EC2 instances. The networking
configuration in EC2, with IPv6 addresses assigned by DHCPv6 and
host egress filtering, makes "duplicate addresses" impossible.
This speeds up the boot process in EC2 by 2 seconds.
Reviewed by: kp, imp, bz
MFC after: 2 weeks
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D33091
This is useful for adding extra packages to the build of an AMI.
For example:
env VM_EXTRA_PACKAGES="zsh" make -C release ec2ami
Approved by: gjb
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
truncate(1) is not case-sensitive with regard to setting the size
of a file. makefs(8), however, does not honor upper-case values.
Update release-specific files and the release(7) manual page to
reflect this.
MFC with: 1ca8842f3a
Submitted by: ehem_freebsd_m5p.com (original)
Differential Review: https://reviews.freebsd.org/D28979
Sponsored by: Rubicon Communications, LLC ("Netgate")
Prior to this commit, EC2 AMIs used a "dual-dhclient" tool which was
launched in place of dhclient and spawned both the base system dhclient
for IPv4 and the ISC dhclient from ports for IPv6.
Now that rtsold supports the "M bit" (managed configuration), we can go
back to having the base system dhclient spawned normally, and provide a
script to rtsold which spawns the ISC dhclient from ports when rtsold
decides that it is appropriate.
Thanks to: bz
MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva
The FreeBSD base system continues to expand. 4GB is now insufficient;
we passed 3 GB in May 2019; we passed 2 GB in August 2017. Over half
of the disk space used is in /usr/lib/debug/.
Without this change, instances boot but are unusable, since the first
thing which breaks when VM filesystems are too small is the "pkg install"
in the VM building process.
This adjusts freebsd-update.conf and portsnap.conf files in EC2 AMIs to
point at the new AWS-hosted mirror network.
Approved by: re (delphij)
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D25498
The ebsnvme-id utility exposes information about EC2 disks -- for
Elastic Block Store volumes, their volume IDs and "linux device
names", and for Instance Store (aka "Ephemeral") disks, their
serial numbers.
The dev_aws_disk rc.d script and associated devd.conf rule maintains
a tree under /dev/aws/disk:
/dev/aws/disk/ebs/<volume ID>
/dev/aws/disk/linuxname/<linux device name>
/dev/aws/disk/ephemeral/<serial number>
which are symlinks to the corresponding nda or nvd devices.
MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva
Since Amazon Elastic File System is only available within AWS, it seems
more appropriate to have this added only in EC2 AMIs rather than
"polluting" non-EC2 images with it.
Reviewed by: gjb
MFC after: 7 days
Relnotes: Amazon EFS filesystems can be automounted by enabling autofs
and placing "/efs -efs" into /etc/auto_master.
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D24791
FreeBSD is in the process of switching from nvd(4) to nda(4) as the disk
device front-end to NVMe. Changing the default in the kernel is tricky
since existing systems may have /dev/nvd* hard-coded e.g. in /etc/fstab;
however, there's no reason to not change the default in HEAD for *new*
systems.
At present I have no intention of MFCing this to stable branches, since
someone might reasonably expect scripts they use for launching and
configuring FreeBSD 12.1 instances to work with FreeBSD 12.2 AMIs, for
example.
Reviewed by: gjb, imp
Relnotes: NVMe disks in EC2 instances launched from 13.0 and later
now show up as nda(4) devices.
Differential Revision: https://reviews.freebsd.org/D24583
I forgot to do this as part of r345858 -- I added it to the
vm_extra_pre_umount in vmimage.subr but forgot that function
was overridden in the EC2 build.
MFC after: 2 weeks
AMIs have been breaking for the past month due to insufficient disk space.
Due to the small amount of overhead in the disk image, the EC2 AMIs end
up with the same (4GB) minimum disk size.
Reported by: Michal Krawczyk
If MACHINE_ARCH doesn't match TARGET_ARCH, and we're not in the special
case of building i386 images on an amd64 host, we need to pull in the
qemu-user-static package; this allows us to run some commands inside
the VM disk image chroot, most notably to install packages.
Reviewed by: gjb
MFC after: 2 weeks
Sponsored by: FreeBSD/EC2 patreon (https://www.patreon.com/cperciva)
This script broke around FreeBSD 11.0 as a result of SWAPMETA no longer
being reported by vmstat -z; but it also needs to be reworked due to the
arrival in EC2 of nvme ephemeral disks.
I'll turn this option back on after I've found time to rewrite the
script in question.
PR: 234686
Reported by: meta@
MFC after: 1 week
the probing and attaching of the PS/2 mouse (not present on EC2) and
keyboard (emulated, but not accessible via EC2).
Note that we disable atkbd0 separately even though during device probing
it shows up as a child of atkbdc0; this is necessary because the device
is also initialized during the early console setup from hammer_time.
This change cuts the kernel boot time on an EC2 c5.4xlarge instance from
7259ms down to 4727 ms.
Approved by: re (marius)
EC2 instances from sysctl.conf to loader.conf; these can all be set as
loader tunables, and setting them in loader.conf gives us the right
behaviour in the event of a kernel panic taking place prior to when
sysctl.conf is processed.
MFC after: 1 week
Since Amazon provides NTP servers within their network, this should
be far superior to using the default NTP pools; and since the service
is provided by Amazon there's very little risk in enabling it by
default. (If someone is able to compromise Amazon's NTP servers and
exploit them to attack EC2 instances, they would almost certainly be
able to compromise EC2 instances even without ntpd running...)
MFC after: 1 week
Relnotes: EC2 instances now keep their clocks synchronized using
the Amazon Time Sync Service (aka. NTP).
EC2 instances are normally launched with an SSH public key specified,
which is then used for logging in (by default, as 'ec2-user'). Having
ChallengeResponseAuthentication enabled (as FreeBSD's default sshd_config
does) has no functional effect in a new EC2 instance, since you can't log
in using a password until a password has been set -- but having this
enabled results in alerts from automated scanning tools which can detect
that sshd advertises support for keyboard-interactive logins (since they
can't detect that accounts have no password set).
EC2 users who want to use passwords to log in to their instances will need
to set 'ChallengeResponseAuthentication yes' in FreeBSD 12.0 and later.
Discussed with: gjb, gtetlow, emaste, des
Requested by: Amazon
X-MFC: No
Relnotes: ChallengeResponseAuthentication is turned off by default in
Amazon EC2 AMIs.
copied from the build host. It is renamed to /etc/resolv.conf.bak
on boot, so never used anyway.
Noticed by: peter
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
immediately available on instances which are running without internet
access (or which can't rely on firstboot_pkgs to install it for some
other reason).
Note that this agent is not enabled by default; to enable it, add
amazon_ssm_agent_enable="YES" to /etc/rc.conf, e.g., by placing the lines
>>/etc/rc.conf
amazon_ssm_agent_enable="YES"
into the EC2 user-data. In addition to being enabled, the agent requires
keys to be provided via IAM Roles; users are encouraged to be very careful
in using this functionality due to the inherent vulnerability in the idea
of providing credentials via a service accessible to any process which can
open an HTTP connection.
Requested by: Amazon
No objection from: re@
Relnotes: FreeBSD/EC2 AMIs now include the Amazon EC2 Systems Manager
(SSM) Agent.
install to prepare an AMI image. This can be used to create a ZFS AMI disk
image using a virtual machine.
Change ec2.conf to use the pkg tool from a chroot rather than trying to
bootstrap it and fail from the livecd readonly filesystem.
Reviewed by: gjb
running on EC2. Due to improvements in EC2, the performance penalty which
was present on some EC2 instances no longer exists, and enabling this
feature now consistently yields ~20% higher throughput with equal or lower
latency.
Reverts: r286063
Approved by: re (gjb)
MFC after: 2 weeks
Relnotes: Improved disk throughput on EC2
issues on some EC2 instance types. Users may want to experiment with
removing this from loader.conf and measuring the performance impact on
the EC2 instances they are using.
