Commit Graph

172 Commits

Author SHA1 Message Date
cperciva
211c3ab0c5 Move debug.{trace,debugger}_on_panic and kern.panic_reboot_wait_time in
EC2 instances from sysctl.conf to loader.conf; these can all be set as
loader tunables, and setting them in loader.conf gives us the right
behaviour in the event of a kernel panic taking place prior to when
sysctl.conf is processed.

MFC after:	1 week
2018-04-18 05:58:27 +00:00
gjb
8f5156d5ad Escape trailing newlines in a long variable list for consistency.
Submitted by:	garga
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-03-26 18:24:16 +00:00
gjb
e62fa66429 Remove google_accounts_manager from VM_RC_LIST in the GCE configuration
file, no longer needed.

PR:		221714
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2018-03-22 17:49:27 +00:00
cperciva
e05626901c Make EC2 instances use Amazon's NTP service for time synchronization.
Since Amazon provides NTP servers within their network, this should
be far superior to using the default NTP pools; and since the service
is provided by Amazon there's very little risk in enabling it by
default.  (If someone is able to compromise Amazon's NTP servers and
exploit them to attack EC2 instances, they would almost certainly be
able to compromise EC2 instances even without ntpd running...)

MFC after:	1 week
Relnotes:	EC2 instances now keep their clocks synchronized using
		the Amazon Time Sync Service (aka. NTP).
2017-12-05 09:22:14 +00:00
cperciva
db1e9749f5 Resurrect r321659: Turn off ChallengeResponseAuthentication for EC2 AMIs.
EC2 instances are normally launched with an SSH public key specified,
which is then used for logging in (by default, as 'ec2-user').  Having
ChallengeResponseAuthentication enabled (as FreeBSD's default sshd_config
does) has no functional effect in a new EC2 instance, since you can't log
in using a password until a password has been set -- but having this
enabled results in alerts from automated scanning tools which can detect
that sshd advertises support for keyboard-interactive logins (since they
can't detect that accounts have no password set).

EC2 users who want to use passwords to log in to their instances will need
to set 'ChallengeResponseAuthentication yes' in FreeBSD 12.0 and later.

Discussed with:	gjb, gtetlow, emaste, des
Requested by:	Amazon
X-MFC:		No
Relnotes:	ChallengeResponseAuthentication is turned off by default in
		Amazon EC2 AMIs.
2017-12-05 09:08:48 +00:00
gjb
421bc6a3f1 Fix an indentation nit.
Sponsored by:	The FreeBSD Foundation
2017-11-30 20:52:01 +00:00
gjb
e54ec1c5ff Remove /etc/resolv.conf from virtual machine images, which is
copied from the build host.  It is renamed to /etc/resolv.conf.bak
on boot, so never used anyway.

Noticed by:	peter
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-11-21 18:02:18 +00:00
imp
433bd38e3a Move sys/boot to stand. Fix all references to new location
Sponsored by:	Netflix
2017-11-14 23:02:19 +00:00
bdrewery
8800d532d3 Add option UNIFIED_OBJDIR, on by default, which moves the default build OBJDIR.
This changes the build OBJDIR from the older style of /usr/obj/<srcdir> for
native builds, and /usr/obj/<target>.<target_arch>/<srcdir> for cross builds to
a new simpler format of /usr/obj/<srcdir>/<target>.<target_arch>.  This
new format is used regardless of cross or native build.  It allows
easier management of multiple source tree object directories.

The UNIFIED_OBJDIR option will be removed and its feature made permanent
for the 12.0 release.

Relnotes:	yes (don't note UNIFIED_OBJDIR option since it will be removed)
Prior work:	D3711 D874
Reviewed by:	gjb, sjg
Discussed at:	https://lists.freebsd.org/pipermail/freebsd-arch/2016-May/017805.html
Discussed with:	emaste
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D12840
2017-11-01 21:22:05 +00:00
cperciva
3d509bf21a Add the amazon-ssm-agent package to EC2 AMI builds. This makes it
immediately available on instances which are running without internet
access (or which can't rely on firstboot_pkgs to install it for some
other reason).

Note that this agent is not enabled by default; to enable it, add
amazon_ssm_agent_enable="YES" to /etc/rc.conf, e.g., by placing the lines
	>>/etc/rc.conf
	amazon_ssm_agent_enable="YES"
into the EC2 user-data.  In addition to being enabled, the agent requires
keys to be provided via IAM Roles; users are encouraged to be very careful
in using this functionality due to the inherent vulnerability in the idea
of providing credentials via a service accessible to any process which can
open an HTTP connection.

Requested by:	Amazon
No objection from:	re@
Relnotes:	FreeBSD/EC2 AMIs now include the Amazon EC2 Systems Manager
		(SSM) Agent.
2017-11-01 00:33:54 +00:00
gjb
e2b3cfc07c Set a default hostname for virtual machine images.
A recent bug in security/sudo causes segmentation faults when
the system is not configured with a hostname, which causes issues
with some virtual machine setups, notably Vagrant.  Set the default
hostname to the output of 'uname -o'.

Submitted by:	Nicholas Fiorentini
Sponsored by:	The FreeBSD Foundation
2017-10-30 13:54:54 +00:00
gjb
47aa838516 Revert r323812 from release/tools/arm.subr, which has broken the
build on arm/armv6 images.

Pointyhat:	gjb (myself)
MFC after:	immediate
MFC note:	releng/10.4 has broken because of this
Sponsored by:	The FreeBSD Foundation
2017-09-22 14:34:27 +00:00
gjb
a2a59746a9 Bootstrap etcupdate(8) and mergemaster(8) databases when creating
virtual machine images and embedded images, similar to what is
done when extracting base.txz to the target root filesystem in
an new installation.

Noticed by:	marius
Tested with:	head@r323729
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-09-20 15:49:12 +00:00
gjb
d0cf5a49b1 Increase the Amazon EC2 AMI image size from 2GB to 3GB to prevent
image build failures due to a full md(4)-backed filesystem.

Sponsored by:	The FreeBSD Foundation
2017-08-28 14:49:26 +00:00
gjb
93241cc840 Use py-google-compute-engine instead for releasing Google Compute
Engine (GCE) images with an updated version of Google's tools.

PR:		221714
Submitted by:	helen _dot_ koike _@_ collabora_dot_com (original)
MFC after:	5 days
Sponsored by:	The FreeBSD Foundation
2017-08-22 15:34:27 +00:00
gjb
de01de65d4 Revert r321659, re-enabling ChallengeResponseAuthentication, which was
discussed a while back between cperciva@ and so@, and I forgot.

Reported by:	cperciva
Sponsored by:	The FreeBSD Foundation
2017-07-28 18:46:02 +00:00
gjb
e893fbefb5 Turn off ChallengeResponseAuthentication for EC2 AMIs, one of EC2's
requirements.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-07-28 18:27:30 +00:00
gjb
47a8c8353f In release/release.sh:
- Rename chroot_arm_armv6_build_release() to chroot_arm_build_release()
  and make it hardware agnostic (such as armv6 -vs- armv7 -vs- arm64).

- Evaluate EMBEDDED_TARGET differently so release/tools/arm.subr can
  be used for arm/armv6 and arm64/aarch64.

- Update comments and copyright.

In release/tools/arm.subr:
- In arm_create_disk(), change the default alignment from 63 to 512k,
  fixing a boot issue on arm64 and EFI. [1]

- Update comments and copyright.

Add a RPI3 configuration file, pieces obtained from Crochet.

Obtained from:	Crochet [1]
MFC after:	5 days
X-MFC-Note:	maybe
Sponsored by:	The FreeBSD Foundation
2017-06-23 00:08:36 +00:00
cperciva
c32675649f Turn on support for the Amazon "Elastic Network Adapter" in EC2 AMIs.
X-MFC-after:	318647 + fixes for some lock ordering warnings
2017-05-25 19:02:54 +00:00
gjb
5f14439e87 Enable DHCP and IPv6 autoconfig on non-cloud VM images.
PR:		203653
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-05-25 12:53:49 +00:00
gjb
3f39f91454 Trim trailing '/release/..' when setting _OBJDIR so arm64/aarch64
boot1.efifat is properly located when creating virtual machine images.

Sponsored by:	The FreeBSD Foundation
2017-04-19 21:18:06 +00:00
thompsa
799a3888fc ec2.conf and vmimage.subr can be used from the installation livecd after
install to prepare an AMI image. This can be used to create a ZFS AMI disk
image using a virtual machine.

Change ec2.conf to use the pkg tool from a chroot rather than trying to
bootstrap it and fail from the livecd readonly filesystem.

Reviewed by:	gjb
2017-03-09 01:26:10 +00:00
gjb
78e869568a Increase the EC2 image size for 12-CURRENT. The recent snapshot
builds of EC2 images for 12-CURRENT failed due to a full filesystem
on the md(4) device during creation.

Sponsored by:	The FreeBSD Foundation
2017-03-02 17:31:59 +00:00
cperciva
1c836fb168 Enable IPv6 networking on Amazon EC2.
MFC after:	1 week
2017-01-15 09:06:45 +00:00
gjb
46f3c9c4a4 - Resize FreeBSD to the size of the OpenStack flavor (growfs).
- Speeds up the boot process by disabling sendmail.
- Allows an user to ssh as root with a public key.
- Make ssh(1) respond faster by disabling DNS lookups.
- Enable DHCP on the vtnet(4) interface.

Note: The CLOUDWARE list has not yet been changed to include the
OpenStack target by default yet.

Submitted by:	Diego Casati
PR:		215258
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2016-12-14 03:01:15 +00:00
lwhsu
72818dba17 - Use virtualbox-ose-additions-nox11 for vagrant image to reduce size
Reviewed by:	brd, gjb, swills
Approved by:	gjb
2016-10-30 15:46:24 +00:00
gjb
00202d1651 Belatedly revert r303119, which was determined to not be
needed.

Sponsored by:	The FreeBSD Foundation
2016-10-24 21:16:21 +00:00
np
fe1964c601 Add the ability to override the size of the swap partition when building
VM images.  The default continues to be 1G.

Reviewed by:	gjb@
Sponsored by:	Chelsio Communications
2016-10-10 22:36:44 +00:00
gjb
c3da6d9f13 Create a /usr/home -> /home symlink for the arm images to
avoid /usr/home confusingly being created as a directory.

Reported by:	Russell Haley
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2016-07-20 22:49:33 +00:00
cperciva
4ff5475d27 Now that potentially buggy versions of Xen are automatically detected
(see r302635), there is no need to force msix interrupt migration off
via loader.conf.

Reverts:	r302184
MFC after:	3 days
2016-07-16 08:04:00 +00:00
cperciva
94125daa17 Enable indirect segment I/Os by default in the Xen blkfront driver when
running on EC2.  Due to improvements in EC2, the performance penalty which
was present on some EC2 instances no longer exists, and enabling this
feature now consistently yields ~20% higher throughput with equal or lower
latency.

Reverts:	r286063
Approved by:	re (gjb)
MFC after:	2 weeks
Relnotes:	Improved disk throughput on EC2
2016-06-29 23:56:24 +00:00
cperciva
9ff7a7dab9 Turn off MSI-X interrupt migration by default in EC2 instances; this works
around a bug in older versions of Xen and unbreaks SR-IOV (aka. "EC2
Enhanced Networking").

Approved by:	re (gjb)
Thanks to:	jhb, Jeremiah Lott
2016-06-24 23:39:44 +00:00
manu
ddb2ec7808 Bump /tmp from 30m to 50m for ARM release images.
30m isn't enough for pkg anymore to extract packagesite.txz.
 40m is fine for now but let's take a safer way as we don't know when pkg will need more.

Reported by:	many
Approved by:	re (gjb), andrew (mentor)
2016-06-16 03:02:27 +00:00
cperciva
1db263cf30 Switch from console="comconsole" to boot_multicons="YES" in EC2.
Amazon recently introduced an API for capturing screenshots of an
emulated VGA device; this commit makes that (somewhat) useful.

MFC after:	3 weeks
2016-06-09 13:58:31 +00:00
gjb
caed01c611 Turn off nfsv4acls for arm/armv6 builds, which has an impact
on performance, especially with SD cards on certain SoCs.

Requested by:	trasz
Discussed with:	ian, kientzle

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2016-06-01 04:18:55 +00:00
gjb
b77609818e Fix tmpfs(5) '/tmp' mount point.
A stray trailing space snuck in with one of the recent
changes, making r290550 and r290573 effectively no-op.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-11-25 19:10:04 +00:00
gjb
44b9d011ea Remove noatime from tmpfs(5) fstab(5) entry.
Submitted by:		Howard Su
MFC after:		3 days
X-MFC-With:		r290550
Sponsored by:	The FreeBSD Foundation
2015-11-09 06:30:26 +00:00
gjb
7320cfca32 Use tmpfs(5) instead of md(4) for '/tmp' mount.
Submitted by:		Nikolai Lifanov
Differential Revision:	D3506
MFC after:		3 days
Sponsored by:	The FreeBSD Foundation
2015-11-08 18:02:39 +00:00
gjb
00e9f99f76 In vm_copy_base(), turn off SU+J on the resultant filesystem,
leaving only SU enabled.

Discussed with:	kib (a few weeks ago)
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-09-29 15:30:27 +00:00
cperciva
84ed76c811 Disable blkif indirect segment I/Os in EC2 by default due to performance
issues on some EC2 instance types.  Users may want to experiment with
removing this from loader.conf and measuring the performance impact on
the EC2 instances they are using.
2015-07-30 04:01:00 +00:00
gjb
70dd4560e3 Fix an out-of-order execution issue regarding pkg(8):
- pkg(8) cannot be removed before subsequent reinvocations
- The PKG_CACHEDIR cannot be cleaned after the repo*.sqlite
  has been removed
- pkg(8) cannot be removed as a precursor to any of the other
  steps involved here

MFC after:	3 days
X-MFC-With:	r285722
X-MFC-Before:	10.2-{BETA3,RC1} (whichever happens next)
Sponsored by:	The FreeBSD Foundation
2015-07-21 03:18:53 +00:00
brd
8f7cde25f2 Add support for building VirtualBox Vagrant images.
Abstract the build, package and upload to handle building either type.

Approved by:	re (gjb)
2015-07-20 16:27:44 +00:00
gjb
eb8d3d79be Do not mount /var/log and /var/tmp as md(4)-backed devices.
Discussed with:	ian
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-07-12 19:17:05 +00:00
gjb
b42f5bb715 Revert r285018, r285019, r285076, r285078, r285082.
The true cause of the missing UFS/MSDOSFS labels has been
identified, and only affects stable/10 at the moment.

An request for commit to stable/10 will be pending RE approval
after this commit.

MFC after:	1 day
X-MFC-Note:	never
X-MFC-Never:	r285018, r285019, r285076, r285078, r285082
Sponsored by:	The FreeBSD Foundation
2015-07-04 00:01:41 +00:00
gjb
5bc952ae8d Test for loader.rc existence before attempting to create
the symlink from loader.rc.sample.

Fix paths relative to the CHROOTDIR.

MFC after:	3 days
X-MFC-With:	r285076, r285078
X-MFC-Before:	10.2-BETA1
Sponsored by:	The FreeBSD Foundation
2015-07-03 06:15:54 +00:00
gjb
f1db22bf1a Revert r285018 and r285019, which attempted to resolve
UFS/MSDOSFS label issues on FreeBSD/arm builds, however
the real problem was addressed in r285076, which is due
to two separate issues, unrelated to md(4) stale device
existence.

MFC after:	3 days
X-MFC-With:	r285076
X-MFC-Before:	10.2-BETA1
Sponsored by:	The FreeBSD Foundation
2015-07-03 06:07:21 +00:00
gjb
33c66012c0 Fix the real problem with missing UFS/MSDOSFS labels on the
FreeBSD/arm builds.  The problem stems from the loader.rc file
not existing, as well as geom_label not being loaded at boot.

For now, add the geom_label_load entry to loader.conf, and
symlink loader.rc.sample to loader.rc, both of which allowed
my BeagleBone Black to boot fine with a UFS label reference in
fstab(5).

MFC after:	3 days
X-MFC-Before:	10.2-BETA1
Sponsored by:	The FreeBSD Foundation
2015-07-03 06:01:37 +00:00
gjb
f2ec46bf54 Fix the path to the UFS/MSDOSFS filesystem labels, since they
cannot possibly exist within the chroot(8) before the target
filesystem actually exists.

MFC after:	3 days
X-MFC-With:	r285018
Sponsored by:	The FreeBSD Foundation
2015-07-02 02:33:58 +00:00
gjb
7f9362f8f8 Implement an evil workaround that prevents UFS/MSDOS labels from being
written to disk with newfs(8) and newfs_msdosfs(8).

When iterating through snapshot builds in serial, it is possible for
a build failure to leave stale md(4) devices behind, in some cases, they
could have a UFS or MSDOS filesystem label assigned.

If the md(4) is not destroyed (or not able to be destroyed, as has
happened recently due to my own fault), the filesystem label that
already exists can interfere with a new md(4) device that is targeted to
have the same label.

This behavior, although admittedly a logic error in the wrapper build
scripts, has caused intermittent reports (in particular with the armv6
builds) of missing UFS/MSDOSFS labels, causing the image to fallback to
the mountroot prompt.  This appears to only happen when the backing
md(4) device is destroyed before the calling umount(8) on the target
mount, after which the UFS/MSDOSFS label persists.

The workaround is this:  If EVERYTHINGISFINE is set to non-empty value,
check for an existing ufs/rootfs and msdosfs/MSDOSBOOT filesystem label
in arm_create_disk(), and rm(1) them if they exist.

The EVERYTHINGISFINE variable is chosen because it is used in exactly
one other place - release/Makefile.mirrors - and there are big scary
warnings at the top of that file as well that it should *not* be used
under normal circumstances.  This should not destroy a build machine
that also uses '/dev/ufs/rootfs' as the UFS label, and I have verified
in extensive local testing that the destroyed label is recreated when
the md(4) is unmounted/mounted, but this really should not be enabled
by anyone.

Having said all that, I absolutely *do* plan MFC this to stable/10 for
the 10.2-RELEASE cycle, as so far, I have only observed this behavior
on stable/10, but this is a temporary solution until I can unravel all
of the failure paths to properly trap them.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-07-02 02:13:20 +00:00
brd
55a27553e8 Add initial support for building Vagrant images for VMWare. Next steps will be adding Virtualbox support and uploading to Hashicorp Atlas for others to consume.
Approved by:	re (gjb)
2015-06-27 20:39:13 +00:00
gjb
72250149ec Add initial support for automatically uploading GCE virtual
machine images to the Google Compute Engine platform.

By default, gcutil/gsutil requires an Oauth2 login generated
from a URL that must be opened in a browser, a verification
code copied back to the terminal from which it was invoked,
etc., etc., making it near impossible for automation.

I've hacked together an evil solution to work around this,
so unless GCE_LOGIN_SKIP is set to a non-empty value, this
Makefile will not do anything useful.

As a result of this commit, remove the gce-package.sh script
that was never, nor will ever be, used.

MFC after:	3 days
X-MFC-Note:	(hopefully)
Sponsored by:	The FreeBSD Foundation
2015-06-27 01:29:59 +00:00
gjb
710db0f35a Ensure ASSUME_ALWAYS_YES is set when deleting pkg(8) from the
DESTDIR, otherwise it can get stuck in ttyin.

This means no 10-STABLE EC2 images this week.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-24 21:30:35 +00:00
gjb
9d1d13b2d4 Chase r284656 and r284658:
Remove the Azure-local vm_extra_create_disk(), since we no longer
need qemu-img to convert the final VHD image to an Azure-compatible
format.

Although the waagent utility is installed from ports, create the
symlink to /usr/sbin, pending investigation on where this is
hard-coded, so it can be reported upstream.  In the meantime, this
is good enough.

MFC after:	3 days
X-MFC-Needs:	r284269, r284270, r284271, r284655,
		r284656, r284657, r284658, r284659
X-MFC-Note:	Required for 10.2-RELEASE, marcel@ has
		implicit approval for the required changes
Sponsored by:	The FreeBSD Foundation
2015-06-21 04:34:57 +00:00
gjb
10fa74d463 Fix path to waagent.
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-19 23:03:05 +00:00
gjb
043d779d9b Work around a potential bug in pw(8) when '-m' is specified.
According to the manual page, '-m' should create the user home
directory, however rigorous testing suggests it does not, and
it is unclear if this is an implementation or expectation issue.

Sponsored by:	The FreeBSD Foundation
2015-06-05 02:22:04 +00:00
gjb
b86973a665 Make sure /home exists within the DESTDIR before creating the
'freebsd' user account.

Sponsored by:	The FreeBSD Foundation
2015-06-04 23:11:24 +00:00
gjb
de010910ed Use the new pw(8) '-R' option to create the users/groups.
MFC after:	1 week
X-MFC-after:	r283961
Sponsored by:	The FreeBSD Foundation
2015-06-03 19:19:25 +00:00
gjb
a82be33c48 Pass the '-b' argument to pw(8) so the '/home' directory is created
in the correct place.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-02 15:02:30 +00:00
gjb
7447f1a3e1 In arm_create_disk(), disable soft updates journaling.
Disabling soft updates journaling appears to resolve issues
with kernel panics, and may also be generally bad to have
enabled for SD cards.

Requested by:	ian
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-01 21:10:47 +00:00
gjb
e961ac3d2e Re-enable arm_create_user(), and pass the '-V DESTDIR/etc' to
pw(8) to set the correct /etc directory for the user/group
files.

Provided by:	ian (thanks!)
MFC after:	3 days
X-MFC-with:	r283894
Sponsored by:	The FreeBSD Foundation
2015-06-01 20:59:18 +00:00
gjb
911994babe Disable arm_create_user(). As-is, this does not create the
user in the userland for the target image, but creates the
user in the build chroot.

Before this is re-enabled, I want to figure out a clean way
to do this without requiring the overhead of third-party
utilities (such as qemu).

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-01 20:26:38 +00:00
gjb
f930e15914 Fix virtual machine disk format creating by passing VMFORMAT
to mkimg(1)

PR:		200068
Submitted by:	Jeremy Norris
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-05-11 14:53:07 +00:00
gjb
fb117627f6 Merge ^/projects/release-arm-redux into ^/head.
Of note:

- This commit adds native FreeBSD/arm release build support without
  requiring out-of-tree utilities.

- Part of this merge removes the WANDBOARD-{SOLO,DUAL,QUAD} kernel
  configuration files, for which the IMX6 kernel configuration file
  should be used instead.

- The resulting images have a 'freebsd' user (password 'freebsd'),
  to allow ssh(1) access when console access is not available (VGA
  or serial).  The default 'root' user password is set to 'root'.

- The /etc/ttys file for arm images now enable both ttyv0 and ttyu0
  by default.

Help from:	many (boot testing, feedback, etc.)
Sponsored by:	The FreeBSD Foundation
2015-05-09 21:08:12 +00:00
gjb
c8fa25fd06 Fix path to /boot/msdos for the target image.
Sponsored by:	The FreeBSD Foundation
2015-05-08 10:38:32 +00:00
gjb
463d845cba Reorder / and /boot/msdos mount to avoid the following error:
Mounting local file systems:
mount: /boot/msdos: No such file or directory

Sponsored by:	The FreeBSD Foundation
2015-05-07 23:46:33 +00:00
gjb
2bf3795a46 Create the /boot/msdos directory so the FAT partition is mounted
properly.

Noticed by:	sbruno
Sponsored by:	The FreeBSD Foundation
2015-05-07 20:25:17 +00:00
gjb
22dede213e Enable the growfs rc(8) script.
Sponsored by:	The FreeBSD Foundation
2015-05-07 18:49:43 +00:00
gjb
3d9f396598 In arm_create_disk(), chroot(8) to the build environment
is necessary.

In arm_install_base(), chroot(8) when installing world
and kernel.  Fix paths for fstab(5) and rc.conf(5).

Sponsored by:	The FreeBSD Foundation
2015-05-07 17:35:48 +00:00
gjb
af0be5ff22 In release.sh, create the /dev directory within the chroot
before attempting to mount(8) devfs.  Also, create the
.OBJDIR for the 'release' target, so files end up in the
correct location.

In tools/arm.subr, fix the target device when creating the
gpart partition scheme.

Sponsored by:	The FreeBSD Foundation
2015-05-07 00:27:28 +00:00
gjb
db697baa50 Add tools/arm.subr to contain common subroutines used for
building arm images.  This is similar to tools/vmimage.subr
used for building virtual machine disk images.  By default,
only arm_create_disk() and arm_install_base() contain real
functionality here, and arm_install_uboot() must be overridden
in the arm/KERNEL.conf file.

In release.sh, make create_arm_armv6_build_release() do
something now.

In arm/BEAGLEBONE.conf, set IMAGE_SIZE, PART_SCHEME, FAT_SIZE,
FAT_TYPE, and MD_ARGS, as well as make arm_install_uboot()
functional.

Parts of this were taken from disecting a previous BEAGLEBONE
image, and other parts obtained from Crochet sources.

Sponsored by:	The FreeBSD Foundation
2015-05-06 19:58:12 +00:00
gjb
1289a25572 Remove crochet configuration files and tools/arm/ directory.
Sponsored by:	The FreeBSD Foundation
2015-05-06 15:59:29 +00:00
gjb
7ac181d7c0 Fix building BEAGLEBONE images with Crochet using the
sysutils/u-boot-beaglebone port:

 - In arm/BEAGLEBONE.conf, set EMBEDDEDPORTS to the
   sysutils/u-boot-beaglebone port.
 - In arm/release.sh, remove BEAGLEBONE from setting WANT_UBOOT
 - In tools/arm/crochet-BEAGLEBONE.conf, override the
   beaglebone_check_uboot(), and set BEAGLEBONE_UBOOT to
   /tmp/external/u-boot-beaglebone, and create symlinks to the
   u-boot files in /usr/local/share/u-boot-beaglebone and the
   uEnv.txt file in crochet/board/Beaglebone/files.

Sponsored by:	The FreeBSD Foundation
2015-05-05 21:08:50 +00:00
gjb
0f66edf8be Add initial support for building RPI2 images.
In release.sh, allow overriding buildenv_setup() before
the handoff to arm/release.sh.

Copy arm/RPI-B.conf -> arm/RPI2.conf, set UBOOT_PORT and
the correct KERNEL, and add the buildenv_setup() override
to install the sysutils/u-boot-rpi2 port/package.

Copy tools/arm/crochet-RPI-B.conf -> tools/arm/crochet-RPI2.conf,
and set the correct entries for the RaspberryPi2 board.

Thanks to:	loos@
Sponsored by:	The FreeBSD Foundation
2015-05-05 16:28:04 +00:00
gjb
e60260c0b0 Fix a comment.
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-04-27 19:49:50 +00:00
gjb
09615c77fe Create the arm64/aarch64 VM disk image as MBR instead of
GPT scheme.  UEFI needs to know the unique partition GUID
with GPT, which changes each time.  Specifically, the QEMU
EFI BIOS file has this hard-coded.[1]

Since the GPT labels are now unavailable, unconditionally
label the root filesystem as 'rootfs' with newfs(8), since
it does not hurt anything anywhere else.  For the arm64 case,
'/' is mounted from /dev/ufs/rootfs; for all other VM images,
'/' is mounted from /dev/gpt/rootfs.

Unfortunately, since the /dev/gpt/swapfs label is also lost,
set NOSWAP=1 for the arm64/aarch64 images.  This is temporary,
until I figure out a scalable solution to this.  But, a certain
piece of softare was written "very fast", and ended up living
for 15 years.  We can deal with this for a week or so.

Information from:	andrew, emaste [1]
Sponsored by:	The FreeBSD Foundation
2015-04-23 01:04:14 +00:00
gjb
9c99303fe2 Revert r281809, which did more harm than good, and apply
a more proper fix when attempting to locate the /boot
files.

Sponsored by:	The FreeBSD Foundation
2015-04-21 14:48:38 +00:00
gjb
62ee754a89 Simplify variable expansion in attempt to fix the vm-image build.
Sponsored by:	The FreeBSD Foundation
2015-04-21 02:49:44 +00:00
gjb
aae7c8c171 Add support for building arm64/aarch64 virtual machine images.
Sponsored by:	The FreeBSD Foundation
2015-04-20 20:06:25 +00:00
gjb
d772346661 When building VM disk images, vm_copy_base() uses tar(1) to
copy the userland from one md(4)-mounted filesystem to a clean
filesystem to prevent remnants of files that were added and
removed from resulting in an unclean filesystem.  When newfs(8)
creates the first filesystem with journaled soft-updates enabled,
the /.sujournal file in the new filesystem cannot be overwritten
by the /.sujournal in the original filesystem.

To avoid this particular error case, do not enable journaled
soft-updates when creating the md(4)-backed filesystems, and
instead use tunefs(8) to enable journaled soft-updates after
the new filesystem is populated in vm_copy_base().

While here, fix a long standing bug where the build environment
/boot files were used by mkimg(1) when creating the VM disk
images by using the files in .OBJDIR.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-04-20 19:54:54 +00:00
cperciva
30e3acf20a It would help if I committed the right patch... 2015-03-31 04:35:35 +00:00
cperciva
2d3f9a1b0b Partially revert r278118 now that the required logic for deciding whether
freebsd-update can be useful has moved into the firstboot_freebsd_update
script.
2015-03-31 01:44:29 +00:00
cperciva
763a23e973 Add bits for building EC2 disk images. Make logic for uploading these
to create EC2 AMIs will come in a later commit.
2015-03-31 01:28:33 +00:00
gjb
b4b14712b0 Sigh. s/AutoSize/Growfs/ following upstream commit r761.
MFH:		3 days
Sponsored by:	The FreeBSD Foundation
2015-03-30 19:08:37 +00:00
cperciva
421e560425 Improve check for whether ${DESTDIR}/dev is mounted.
Submitted by:	gcooper
2015-03-30 09:08:24 +00:00
cperciva
e41362f7d1 Clean up filesystem unmounting in vmimage builds:
* Remove vm_umount_base function which is currently unused.
* Add umount_loop function which loops attempting to unmount one filesystem.
* Replace calls to umount with calls to umount_loop.
* Don't attempt to unmount ${DESTDIR}/dev if it isn't mounted.

The looping is necessary because sometimes umount fails due to filesystems
being busy.  The most common cause of such busyness is periodic(8) jobs
running `find / ...`.

Reviewed by:	gjb
2015-03-30 08:33:19 +00:00
cperciva
f1bd255f86 When creating VM images, copy the contents of the created filesystem into
a new filesystem before packaging it into a disk image.  This prevents
"remnants" of deleted files from showing up in the VM images, and reduces
their compressed size (by about 10% for the cloudware images) as a result.

Looks good to:	gjb
2015-03-20 19:40:19 +00:00
gjb
1a0d38818e Add a CLOUDPACKAGE variable to contain a list of targets for
post-install packaging cloud provider images.

Add a 'gce-package.sh' script to generate the final output
image ready for upload to the GCE platform.  Right now, this
is the only image that has a specific output format (GNU-tar),
and this implementation is expected to be temporary.

This is not directly connected to the other release targets.

MFC after:	1 week
X-MFC-with:	r279249, r279250
Sponsored by:	The FreeBSD Foundation
2015-02-24 22:27:02 +00:00
gjb
2c9ed945e2 Use ifconfig_DEFAULT="" instead of hard-coding vtnet(4)
MFC after:	2 weeks
X-MFC-with:	r277458, r277536, r277606, r277609,
		r277836, r278118
Sponsored by:	The FreeBSD Foundation
2015-02-03 00:10:48 +00:00
gjb
103061b4c4 Move service_enable="YES" rc.conf(5) additions to
VM_RC_LIST and deduplicate.

Evaluate if firstboot_freebsd_update should be enabled
based on UNAME_r, because it is not supported for the
-CURRENT or -STABLE branches.

MFC after:	2 weeks
X-MFC-with:	r277458, r277536, r277606, r277609,
		r277836
Sponsored by:	The FreeBSD Foundation
2015-02-03 00:07:54 +00:00
gjb
bb3fec85f3 Merge the following revisions from ^/projects/release-vmimage:
r273823-r273826, r273833, r273836, r273944, r274069-r274071,
 r274134, r274211, r274280-r274285, r274287-r274288, r274292,
 r274296-r274297, r274356, r274533, r274725, r274726, r274729,
 r274734, r274771, r274945-r274946, r277180, r277183-r277184,
 r277186-r277187, r277250-r277253, r277263-r277264, r277383-r277384,
 r277393-r277395, r277438-r277439, r277447, r277455:

 r273823:
  Move virtual machine / cloud provider targets and
  options from release/Makefile to their own Makefile.

 r273824:
  Add glue to allow enabling building cloud provider VM images
  by default.

  When WITH_CLOUDWARE is not empty, add CLOUDTARGETS to the
  release/Makefile 'release' target.

 r273825:
  Avoid hard-coding the Azure image file format.  While here,
  avoid using OSRELEASE for the output file name.

 r273826:
  Remove a few vestiges of passing an exit code to panic().

 r273833:
  Initial commit providing a mechanism to create openstack images
  as part of the release build.

 r273836:
  Fix output file name for openstack images.  No further conversion
  is necessary for this VM file target, so there is no need to append
  the '.raw' suffix here.

 r273944:
  Uncomment the cloudinit rc.conf(5) line.

 r274069:
  Add line continuation so OPENSTACKCONF is actually included in the env(1).

 r274070:
  Add a 'vm-cloudware' target, used to drive all targets in CLOUDTARGETS.

 r274071:
  Add examples for WITH_CLOUDWARE to release.conf.sample.
  Add WITH_CLOUDWARE evaluation to RELEASE_RMAKEFLAGS.

 r274134:
  Initial rewrite to consolidate VM image build scripts into one.

 r274211:
  Add write_partition_layout() used to populate the final image.

  Fix duplicated mkimg(1) call in vm_create_disk().

  Add primitive (untested) PowerPC/PowerPC64 VM image support.

  Note: As it is currently written, the /boot/pmbr and
  /boot/{gptboot,boot1.hfs} use the build host and not the target
  build.  Fixing this is likely going to be a hack in itself.

 r274280:
  Return if vm_create_disk() is unsuccessful.

 r274281:
  Add CLEANFILES entry for VM targets

 r274282:
  Add vm_extra_pre_umount() prototype to vmimage.subr.

 r274283:
  Fix DESTDIR for installworld, and make sure it is created before use.

 r274284:
  Move usage() from vmimage.subr to mk-vmimage.sh, in case vmimage.subr
  has not been sourced.

 r274285:
  Spell 'OPTARG' correctly.  Actually call vm_create_base().

 r274287:
  Fix line continuation in write_partition_layout().
  Remove variable test that is no longer needed.

 r274288:
  Fix scheme flag to mkimg(1).

 r274292:
  mount(8) and umount(8) devfs(5) as needed.

 r274296:
  Change path for mk-vmimage.sh from ${TARGET}/ to scripts/ now that
  it is consolidated into one file.

  Fix paths for the base image and output disk image files.

 r274297:
  Call cleanup() after everything is done.

 r274356:
  Remove a stray directory from CLEANFILES.

 r274533:
  Set the boot partition type to 'apple-boot' for powerpc.

 r274725:
  In vm_install_base(), copy the host resolv.conf into
  the build chroot before attempting to do anything that
  requires working DNS (i.e., pkg bootstrap).

  In vm_extra_pre_umount(), remove the resolv.conf before
  the disk image is unmounted from the backing md(4).

 r274726 (cperciva):
  Silence errors when umounting the chroot's /dev, since it
  probably doesn't exist when we're running this.

  Unmount filesystems before attempting to destroy the md which
  holds them.

 r274729 (cperciva):
  Unmount filesystem and destroy md before we read the vnode from
  disk and package it into a disk image.  Otherwise we end up
  packaging an unclean filesystem.

 r274734 (cperciva):
  Merge duplicative vm-CLOUDTYPE targets before additional duplication
  gets added by the impending arrival of ec2 and gcloud.

 r274771 (cperciva):
  Add NOSWAP option which can be set by a vmimage.conf file to specify
  that no swap space should be created in the image.  This will be used
  by EC2 builds, since FreeBSD/EC2 allocates swap space on "ephemeral"
  disks which are physically attached to the Xen host node.

 r274945:
  In vm_extra_install_packages(), only bootstrap pkg(8) if
  VM_EXTRA_PACKAGES is empty.

  In vm_extra_pre_umount(), cleanup downloaded packages if pkg(8) was
  bootstrapped earlier.

 r274946:
  Fix indentation nit.

 r277180:
  In vm_extra_install_base(), do not install waagent in the openstack
  image, because it is not used.  This appears to be a copy mistake.

  Remove vm_extra_install_base() from the openstack.conf entirely,
  since it does not need to be overridden.

 r277183:
  Enable the textmode console by default for VM images, since there is
  no way to tell if the environment will be able to use the
  graphics-mode console.

 r277184:
  Enable password-less sudo for openstack images.

 r277186:
  Update the VM_EXTRA_PACKAGES list for the openstack images.

  The documentation suggests doing a "just fetch this and run it"-style
  bootstrap, from which the list of dependencies was obtained (in
  github, at: pellaeon/bsd-cloudinit-installer)

  There is one Python dependency unmet, oslo.config, which is not in
  the Ports Collection.

 r277187:
  Add a comment to note that setting hw.vga.textmode=1 is temporary.

 r277250:
  Remove vm_extra_install_base() for the Azure image, now that the
  waagent exists in the ports tree.

  Add sysutils/azure-agent to the VM_EXTRA_PACKAGES list.

  In vm_extra_pre_umount(), remove the explicit pkg(8) install
  list, as dependencies are resolved by sysutils/azure-agent.

 r277251:
  Add a 'list-cloudware' target to print the list of supported CLOUDWARE
  values and a description.

  Add the AZURE_DESC and OPENSTACK_DESC descriptions.

 r277252:
  Update release(7)

 r277253:
  Add 'list-vmtargets' target, which produces a list of all supported
  VM and cloud provider images.

  Add VHD_DESC, VMDK_DESC, QCOW2_DESC, RAW_DESC image descriptions.

  Format the output to make a bit more readable.

  Update release(7) to document the list-vmtargets target.

 r277263:
  Add initial support for the GCE (Google Compute Engine) cloud hosting
  provider image.

 r277264:
  Style and line length cleanup.

 r277383:
  Remove the console setting from rc.conf(5), which is not used there.
  While here, set console to include vidconsole in the loader.conf(5).

 r277384:
  Fix an indentation nit.
  No functional changes.

 r277393:
  Remove the pkg-clean(8) call from vm_extra_pre_umount() since the
  function is often overridden.

  Add vm_extra_pkg_rmcache() to call pkg-clean(8) to avoid duplicated
  code.

 r277394:
  Move resolv.conf(5) removal back to vm_extra_pre_umount() where it
  belongs.

  The GCE image needs resolv.conf(5) to exist (created as part of the
  image setup), so it cannot be removed.

 r277395:
  Comment the line that configures ttys(5) to 'off', which makes it
  impossible to test that the image boots.

  Add a note explaining why the line is commented, and not (yet) removed
  entirely.

 r277438:
  Move the 'install' bits that are specific to virtual machine images
  from the Makefile to Makefile.vm.

  Rename the 'install' target to 'release-install', and add a new
  'vm-install' target.

  Add a new 'install' target that invokes the new targets.

 r277439:
  Add WITH_CLOUDWARE to the list of make(1) variables for the release
  build.

 r277447:
  Remove hw.vga.textmode=1 from the VM image loader.conf, which was
  included during test builds and not intended to be included when
  merging this project branch back to head.

 r277455:
  Remove mk-azure.sh, which is no longer needed.

MFC after:	1 month
X-MFC-To:	stable/10 (requires mkimg(1))
Help from:	cperciva, swills
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2015-01-20 23:56:04 +00:00
gjb
2f6be49b3a Add initial support for the GCE (Google Compute Engine)
cloud hosting provider image.

Many thanks to swills@ for his work on getting this to
this point.

Submitted by:	swills
Sponsored by:	The FreeBSD Foundation
2015-01-16 19:27:19 +00:00
gjb
0d86658a18 Remove vm_extra_install_base() for the Azure image, now
that the waagent exists in the ports tree.

Add sysutils/azure-agent to the VM_EXTRA_PACKAGES list.

In vm_extra_pre_umount(), remove the explicit pkg(8) install
list, as dependencies are resolved by sysutils/azure-agent.

Sponsored by:	The FreeBSD Foundation
2015-01-16 15:37:07 +00:00
gjb
8bf2cea4c9 Add a comment to note that setting hw.vga.textmode=1 is
temporary.

Sponsored by:	The FreeBSD Foundation
2015-01-14 16:42:54 +00:00
gjb
57dffb5c40 Update the VM_EXTRA_PACKAGES list for the openstack
images.

The documentation suggests doing a "just fetch this
and run it"-style bootstrap, from which the list of
dependencies was obtained (in github, at:
pellaeon/bsd-cloudinit-installer)

There is one Python dependency unmet, oslo.config,
which is not in the Ports Collection.

Sponsored by:	The FreeBSD Foundation
2015-01-14 16:27:43 +00:00
gjb
64286bf832 Enable password-less sudo for openstack images.
Sponsored by:	The FreeBSD Foundation
2015-01-14 15:45:18 +00:00
gjb
5107643b94 Enable the textmode console by default for VM images,
since there is no way to tell if the environment will
be able to use the graphics-mode console.

Sponsored by:	The FreeBSD Foundation
2015-01-14 15:23:58 +00:00
gjb
f0d9ad0c85 In vm_extra_install_base(), do not install waagent
in the openstack image, because it is not used.  This
appears to be a copy mistake.

Remove vm_extra_install_base() from the openstack.conf
entirely, since it does not need to be overridden.

Sponsored by:	The FreeBSD Foundation
2015-01-14 14:49:05 +00:00
gjb
7f08d7ffb1 Fix indentation nit.
Sponsored by:	The FreeBSD Foundation
2014-11-24 02:36:43 +00:00
gjb
14b8fb2398 In vm_extra_install_packages(), only bootstrap pkg(8)
if VM_EXTRA_PACKAGES is empty.

In vm_extra_pre_umount(), cleanup downloaded packages
if pkg(8) was bootstrapped earlier.

Inspired by:	cperciva
Sponsored by:	The FreeBSD Foundation
2014-11-24 02:34:01 +00:00
gjb
5074f8a73e Revert r274773, after I *specifically* objected to this change
numerous times.

Sponsored by:	The FreeBSD Foundation
2014-11-21 02:30:37 +00:00