Commit Graph

125 Commits

Author SHA1 Message Date
Colin Percival
4a93691064 Make EC2 instances use Amazon's NTP service for time synchronization.
Since Amazon provides NTP servers within their network, this should
be far superior to using the default NTP pools; and since the service
is provided by Amazon there's very little risk in enabling it by
default.  (If someone is able to compromise Amazon's NTP servers and
exploit them to attack EC2 instances, they would almost certainly be
able to compromise EC2 instances even without ntpd running...)

MFC after:	1 week
Relnotes:	EC2 instances now keep their clocks synchronized using
		the Amazon Time Sync Service (aka. NTP).
2017-12-05 09:22:14 +00:00
Colin Percival
4ba35bc4db Resurrect r321659: Turn off ChallengeResponseAuthentication for EC2 AMIs.
EC2 instances are normally launched with an SSH public key specified,
which is then used for logging in (by default, as 'ec2-user').  Having
ChallengeResponseAuthentication enabled (as FreeBSD's default sshd_config
does) has no functional effect in a new EC2 instance, since you can't log
in using a password until a password has been set -- but having this
enabled results in alerts from automated scanning tools which can detect
that sshd advertises support for keyboard-interactive logins (since they
can't detect that accounts have no password set).

EC2 users who want to use passwords to log in to their instances will need
to set 'ChallengeResponseAuthentication yes' in FreeBSD 12.0 and later.

Discussed with:	gjb, gtetlow, emaste, des
Requested by:	Amazon
X-MFC:		No
Relnotes:	ChallengeResponseAuthentication is turned off by default in
		Amazon EC2 AMIs.
2017-12-05 09:08:48 +00:00
Glen Barber
c205468e00 Fix an indentation nit.
Sponsored by:	The FreeBSD Foundation
2017-11-30 20:52:01 +00:00
Glen Barber
3103eac6a0 Remove /etc/resolv.conf from virtual machine images, which is
copied from the build host.  It is renamed to /etc/resolv.conf.bak
on boot, so never used anyway.

Noticed by:	peter
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-11-21 18:02:18 +00:00
Warner Losh
ca987d4641 Move sys/boot to stand. Fix all references to new location
Sponsored by:	Netflix
2017-11-14 23:02:19 +00:00
Bryan Drewery
dfa099890c Add option UNIFIED_OBJDIR, on by default, which moves the default build OBJDIR.
This changes the build OBJDIR from the older style of /usr/obj/<srcdir> for
native builds, and /usr/obj/<target>.<target_arch>/<srcdir> for cross builds to
a new simpler format of /usr/obj/<srcdir>/<target>.<target_arch>.  This
new format is used regardless of cross or native build.  It allows
easier management of multiple source tree object directories.

The UNIFIED_OBJDIR option will be removed and its feature made permanent
for the 12.0 release.

Relnotes:	yes (don't note UNIFIED_OBJDIR option since it will be removed)
Prior work:	D3711 D874
Reviewed by:	gjb, sjg
Discussed at:	https://lists.freebsd.org/pipermail/freebsd-arch/2016-May/017805.html
Discussed with:	emaste
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D12840
2017-11-01 21:22:05 +00:00
Colin Percival
9d98ae7d5f Add the amazon-ssm-agent package to EC2 AMI builds. This makes it
immediately available on instances which are running without internet
access (or which can't rely on firstboot_pkgs to install it for some
other reason).

Note that this agent is not enabled by default; to enable it, add
amazon_ssm_agent_enable="YES" to /etc/rc.conf, e.g., by placing the lines
	>>/etc/rc.conf
	amazon_ssm_agent_enable="YES"
into the EC2 user-data.  In addition to being enabled, the agent requires
keys to be provided via IAM Roles; users are encouraged to be very careful
in using this functionality due to the inherent vulnerability in the idea
of providing credentials via a service accessible to any process which can
open an HTTP connection.

Requested by:	Amazon
No objection from:	re@
Relnotes:	FreeBSD/EC2 AMIs now include the Amazon EC2 Systems Manager
		(SSM) Agent.
2017-11-01 00:33:54 +00:00
Glen Barber
2170fff8a5 Set a default hostname for virtual machine images.
A recent bug in security/sudo causes segmentation faults when
the system is not configured with a hostname, which causes issues
with some virtual machine setups, notably Vagrant.  Set the default
hostname to the output of 'uname -o'.

Submitted by:	Nicholas Fiorentini
Sponsored by:	The FreeBSD Foundation
2017-10-30 13:54:54 +00:00
Glen Barber
3f04d8c304 Revert r323812 from release/tools/arm.subr, which has broken the
build on arm/armv6 images.

Pointyhat:	gjb (myself)
MFC after:	immediate
MFC note:	releng/10.4 has broken because of this
Sponsored by:	The FreeBSD Foundation
2017-09-22 14:34:27 +00:00
Glen Barber
c52962c68e Bootstrap etcupdate(8) and mergemaster(8) databases when creating
virtual machine images and embedded images, similar to what is
done when extracting base.txz to the target root filesystem in
an new installation.

Noticed by:	marius
Tested with:	head@r323729
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-09-20 15:49:12 +00:00
Glen Barber
505f20a67b Increase the Amazon EC2 AMI image size from 2GB to 3GB to prevent
image build failures due to a full md(4)-backed filesystem.

Sponsored by:	The FreeBSD Foundation
2017-08-28 14:49:26 +00:00
Glen Barber
83ffbdb133 Use py-google-compute-engine instead for releasing Google Compute
Engine (GCE) images with an updated version of Google's tools.

PR:		221714
Submitted by:	helen _dot_ koike _@_ collabora_dot_com (original)
MFC after:	5 days
Sponsored by:	The FreeBSD Foundation
2017-08-22 15:34:27 +00:00
Glen Barber
b200285a05 Revert r321659, re-enabling ChallengeResponseAuthentication, which was
discussed a while back between cperciva@ and so@, and I forgot.

Reported by:	cperciva
Sponsored by:	The FreeBSD Foundation
2017-07-28 18:46:02 +00:00
Glen Barber
5941ae31e0 Turn off ChallengeResponseAuthentication for EC2 AMIs, one of EC2's
requirements.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-07-28 18:27:30 +00:00
Glen Barber
751c20d8a5 In release/release.sh:
- Rename chroot_arm_armv6_build_release() to chroot_arm_build_release()
  and make it hardware agnostic (such as armv6 -vs- armv7 -vs- arm64).

- Evaluate EMBEDDED_TARGET differently so release/tools/arm.subr can
  be used for arm/armv6 and arm64/aarch64.

- Update comments and copyright.

In release/tools/arm.subr:
- In arm_create_disk(), change the default alignment from 63 to 512k,
  fixing a boot issue on arm64 and EFI. [1]

- Update comments and copyright.

Add a RPI3 configuration file, pieces obtained from Crochet.

Obtained from:	Crochet [1]
MFC after:	5 days
X-MFC-Note:	maybe
Sponsored by:	The FreeBSD Foundation
2017-06-23 00:08:36 +00:00
Colin Percival
0edcc5b8d5 Turn on support for the Amazon "Elastic Network Adapter" in EC2 AMIs.
X-MFC-after:	318647 + fixes for some lock ordering warnings
2017-05-25 19:02:54 +00:00
Glen Barber
f8e26d73e1 Enable DHCP and IPv6 autoconfig on non-cloud VM images.
PR:		203653
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-05-25 12:53:49 +00:00
Glen Barber
f56769dd00 Trim trailing '/release/..' when setting _OBJDIR so arm64/aarch64
boot1.efifat is properly located when creating virtual machine images.

Sponsored by:	The FreeBSD Foundation
2017-04-19 21:18:06 +00:00
Andrew Thompson
ad33b77611 ec2.conf and vmimage.subr can be used from the installation livecd after
install to prepare an AMI image. This can be used to create a ZFS AMI disk
image using a virtual machine.

Change ec2.conf to use the pkg tool from a chroot rather than trying to
bootstrap it and fail from the livecd readonly filesystem.

Reviewed by:	gjb
2017-03-09 01:26:10 +00:00
Glen Barber
a744f4edcd Increase the EC2 image size for 12-CURRENT. The recent snapshot
builds of EC2 images for 12-CURRENT failed due to a full filesystem
on the md(4) device during creation.

Sponsored by:	The FreeBSD Foundation
2017-03-02 17:31:59 +00:00
Colin Percival
9763fbf7ef Enable IPv6 networking on Amazon EC2.
MFC after:	1 week
2017-01-15 09:06:45 +00:00
Glen Barber
82fe834050 - Resize FreeBSD to the size of the OpenStack flavor (growfs).
- Speeds up the boot process by disabling sendmail.
- Allows an user to ssh as root with a public key.
- Make ssh(1) respond faster by disabling DNS lookups.
- Enable DHCP on the vtnet(4) interface.

Note: The CLOUDWARE list has not yet been changed to include the
OpenStack target by default yet.

Submitted by:	Diego Casati
PR:		215258
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-12-14 03:01:15 +00:00
Li-Wen Hsu
49b53bee5a - Use virtualbox-ose-additions-nox11 for vagrant image to reduce size
Reviewed by:	brd, gjb, swills
Approved by:	gjb
2016-10-30 15:46:24 +00:00
Glen Barber
f5355fadf7 Belatedly revert r303119, which was determined to not be
needed.

Sponsored by:	The FreeBSD Foundation
2016-10-24 21:16:21 +00:00
Navdeep Parhar
eb46c98711 Add the ability to override the size of the swap partition when building
VM images.  The default continues to be 1G.

Reviewed by:	gjb@
Sponsored by:	Chelsio Communications
2016-10-10 22:36:44 +00:00
Glen Barber
d82d5ebec6 Create a /usr/home -> /home symlink for the arm images to
avoid /usr/home confusingly being created as a directory.

Reported by:	Russell Haley
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2016-07-20 22:49:33 +00:00
Colin Percival
d53ee33e1c Now that potentially buggy versions of Xen are automatically detected
(see r302635), there is no need to force msix interrupt migration off
via loader.conf.

Reverts:	r302184
MFC after:	3 days
2016-07-16 08:04:00 +00:00
Colin Percival
4d1afa14d5 Enable indirect segment I/Os by default in the Xen blkfront driver when
running on EC2.  Due to improvements in EC2, the performance penalty which
was present on some EC2 instances no longer exists, and enabling this
feature now consistently yields ~20% higher throughput with equal or lower
latency.

Reverts:	r286063
Approved by:	re (gjb)
MFC after:	2 weeks
Relnotes:	Improved disk throughput on EC2
2016-06-29 23:56:24 +00:00
Colin Percival
60406292e4 Turn off MSI-X interrupt migration by default in EC2 instances; this works
around a bug in older versions of Xen and unbreaks SR-IOV (aka. "EC2
Enhanced Networking").

Approved by:	re (gjb)
Thanks to:	jhb, Jeremiah Lott
2016-06-24 23:39:44 +00:00
Emmanuel Vadot
c7161c4adc Bump /tmp from 30m to 50m for ARM release images.
30m isn't enough for pkg anymore to extract packagesite.txz.
 40m is fine for now but let's take a safer way as we don't know when pkg will need more.

Reported by:	many
Approved by:	re (gjb), andrew (mentor)
2016-06-16 03:02:27 +00:00
Colin Percival
f207bd78e6 Switch from console="comconsole" to boot_multicons="YES" in EC2.
Amazon recently introduced an API for capturing screenshots of an
emulated VGA device; this commit makes that (somewhat) useful.

MFC after:	3 weeks
2016-06-09 13:58:31 +00:00
Glen Barber
a99eb3157e Turn off nfsv4acls for arm/armv6 builds, which has an impact
on performance, especially with SD cards on certain SoCs.

Requested by:	trasz
Discussed with:	ian, kientzle

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2016-06-01 04:18:55 +00:00
Glen Barber
a9bf704474 Fix tmpfs(5) '/tmp' mount point.
A stray trailing space snuck in with one of the recent
changes, making r290550 and r290573 effectively no-op.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-11-25 19:10:04 +00:00
Glen Barber
0eea47e68b Remove noatime from tmpfs(5) fstab(5) entry.
Submitted by:		Howard Su
MFC after:		3 days
X-MFC-With:		r290550
Sponsored by:	The FreeBSD Foundation
2015-11-09 06:30:26 +00:00
Glen Barber
6a833a7a2c Use tmpfs(5) instead of md(4) for '/tmp' mount.
Submitted by:		Nikolai Lifanov
Differential Revision:	D3506
MFC after:		3 days
Sponsored by:	The FreeBSD Foundation
2015-11-08 18:02:39 +00:00
Glen Barber
91c42fff77 In vm_copy_base(), turn off SU+J on the resultant filesystem,
leaving only SU enabled.

Discussed with:	kib (a few weeks ago)
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-09-29 15:30:27 +00:00
Colin Percival
a5c1653b5c Disable blkif indirect segment I/Os in EC2 by default due to performance
issues on some EC2 instance types.  Users may want to experiment with
removing this from loader.conf and measuring the performance impact on
the EC2 instances they are using.
2015-07-30 04:01:00 +00:00
Glen Barber
6a277d6e8c Fix an out-of-order execution issue regarding pkg(8):
- pkg(8) cannot be removed before subsequent reinvocations
- The PKG_CACHEDIR cannot be cleaned after the repo*.sqlite
  has been removed
- pkg(8) cannot be removed as a precursor to any of the other
  steps involved here

MFC after:	3 days
X-MFC-With:	r285722
X-MFC-Before:	10.2-{BETA3,RC1} (whichever happens next)
Sponsored by:	The FreeBSD Foundation
2015-07-21 03:18:53 +00:00
Brad Davis
a54bd5953b Add support for building VirtualBox Vagrant images.
Abstract the build, package and upload to handle building either type.

Approved by:	re (gjb)
2015-07-20 16:27:44 +00:00
Glen Barber
a1912cc902 Do not mount /var/log and /var/tmp as md(4)-backed devices.
Discussed with:	ian
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-07-12 19:17:05 +00:00
Glen Barber
a1c0309edc Revert r285018, r285019, r285076, r285078, r285082.
The true cause of the missing UFS/MSDOSFS labels has been
identified, and only affects stable/10 at the moment.

An request for commit to stable/10 will be pending RE approval
after this commit.

MFC after:	1 day
X-MFC-Note:	never
X-MFC-Never:	r285018, r285019, r285076, r285078, r285082
Sponsored by:	The FreeBSD Foundation
2015-07-04 00:01:41 +00:00
Glen Barber
ef3a5d20dd Test for loader.rc existence before attempting to create
the symlink from loader.rc.sample.

Fix paths relative to the CHROOTDIR.

MFC after:	3 days
X-MFC-With:	r285076, r285078
X-MFC-Before:	10.2-BETA1
Sponsored by:	The FreeBSD Foundation
2015-07-03 06:15:54 +00:00
Glen Barber
174fe0cfce Revert r285018 and r285019, which attempted to resolve
UFS/MSDOSFS label issues on FreeBSD/arm builds, however
the real problem was addressed in r285076, which is due
to two separate issues, unrelated to md(4) stale device
existence.

MFC after:	3 days
X-MFC-With:	r285076
X-MFC-Before:	10.2-BETA1
Sponsored by:	The FreeBSD Foundation
2015-07-03 06:07:21 +00:00
Glen Barber
a6d3c7cdd5 Fix the real problem with missing UFS/MSDOSFS labels on the
FreeBSD/arm builds.  The problem stems from the loader.rc file
not existing, as well as geom_label not being loaded at boot.

For now, add the geom_label_load entry to loader.conf, and
symlink loader.rc.sample to loader.rc, both of which allowed
my BeagleBone Black to boot fine with a UFS label reference in
fstab(5).

MFC after:	3 days
X-MFC-Before:	10.2-BETA1
Sponsored by:	The FreeBSD Foundation
2015-07-03 06:01:37 +00:00
Glen Barber
e8d7333d7d Fix the path to the UFS/MSDOSFS filesystem labels, since they
cannot possibly exist within the chroot(8) before the target
filesystem actually exists.

MFC after:	3 days
X-MFC-With:	r285018
Sponsored by:	The FreeBSD Foundation
2015-07-02 02:33:58 +00:00
Glen Barber
e07ca0423d Implement an evil workaround that prevents UFS/MSDOS labels from being
written to disk with newfs(8) and newfs_msdosfs(8).

When iterating through snapshot builds in serial, it is possible for
a build failure to leave stale md(4) devices behind, in some cases, they
could have a UFS or MSDOS filesystem label assigned.

If the md(4) is not destroyed (or not able to be destroyed, as has
happened recently due to my own fault), the filesystem label that
already exists can interfere with a new md(4) device that is targeted to
have the same label.

This behavior, although admittedly a logic error in the wrapper build
scripts, has caused intermittent reports (in particular with the armv6
builds) of missing UFS/MSDOSFS labels, causing the image to fallback to
the mountroot prompt.  This appears to only happen when the backing
md(4) device is destroyed before the calling umount(8) on the target
mount, after which the UFS/MSDOSFS label persists.

The workaround is this:  If EVERYTHINGISFINE is set to non-empty value,
check for an existing ufs/rootfs and msdosfs/MSDOSBOOT filesystem label
in arm_create_disk(), and rm(1) them if they exist.

The EVERYTHINGISFINE variable is chosen because it is used in exactly
one other place - release/Makefile.mirrors - and there are big scary
warnings at the top of that file as well that it should *not* be used
under normal circumstances.  This should not destroy a build machine
that also uses '/dev/ufs/rootfs' as the UFS label, and I have verified
in extensive local testing that the destroyed label is recreated when
the md(4) is unmounted/mounted, but this really should not be enabled
by anyone.

Having said all that, I absolutely *do* plan MFC this to stable/10 for
the 10.2-RELEASE cycle, as so far, I have only observed this behavior
on stable/10, but this is a temporary solution until I can unravel all
of the failure paths to properly trap them.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-07-02 02:13:20 +00:00
Brad Davis
41f7db3a4f Add initial support for building Vagrant images for VMWare. Next steps will be adding Virtualbox support and uploading to Hashicorp Atlas for others to consume.
Approved by:	re (gjb)
2015-06-27 20:39:13 +00:00
Glen Barber
eb64cd58f5 Add initial support for automatically uploading GCE virtual
machine images to the Google Compute Engine platform.

By default, gcutil/gsutil requires an Oauth2 login generated
from a URL that must be opened in a browser, a verification
code copied back to the terminal from which it was invoked,
etc., etc., making it near impossible for automation.

I've hacked together an evil solution to work around this,
so unless GCE_LOGIN_SKIP is set to a non-empty value, this
Makefile will not do anything useful.

As a result of this commit, remove the gce-package.sh script
that was never, nor will ever be, used.

MFC after:	3 days
X-MFC-Note:	(hopefully)
Sponsored by:	The FreeBSD Foundation
2015-06-27 01:29:59 +00:00
Glen Barber
67076e4d58 Ensure ASSUME_ALWAYS_YES is set when deleting pkg(8) from the
DESTDIR, otherwise it can get stuck in ttyin.

This means no 10-STABLE EC2 images this week.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-24 21:30:35 +00:00
Glen Barber
798e3ec502 Chase r284656 and r284658:
Remove the Azure-local vm_extra_create_disk(), since we no longer
need qemu-img to convert the final VHD image to an Azure-compatible
format.

Although the waagent utility is installed from ports, create the
symlink to /usr/sbin, pending investigation on where this is
hard-coded, so it can be reported upstream.  In the meantime, this
is good enough.

MFC after:	3 days
X-MFC-Needs:	r284269, r284270, r284271, r284655,
		r284656, r284657, r284658, r284659
X-MFC-Note:	Required for 10.2-RELEASE, marcel@ has
		implicit approval for the required changes
Sponsored by:	The FreeBSD Foundation
2015-06-21 04:34:57 +00:00