Commit Graph

90 Commits

Author SHA1 Message Date
Glen Barber
91c42fff77 In vm_copy_base(), turn off SU+J on the resultant filesystem,
leaving only SU enabled.

Discussed with:	kib (a few weeks ago)
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-09-29 15:30:27 +00:00
Colin Percival
a5c1653b5c Disable blkif indirect segment I/Os in EC2 by default due to performance
issues on some EC2 instance types.  Users may want to experiment with
removing this from loader.conf and measuring the performance impact on
the EC2 instances they are using.
2015-07-30 04:01:00 +00:00
Glen Barber
6a277d6e8c Fix an out-of-order execution issue regarding pkg(8):
- pkg(8) cannot be removed before subsequent reinvocations
- The PKG_CACHEDIR cannot be cleaned after the repo*.sqlite
  has been removed
- pkg(8) cannot be removed as a precursor to any of the other
  steps involved here

MFC after:	3 days
X-MFC-With:	r285722
X-MFC-Before:	10.2-{BETA3,RC1} (whichever happens next)
Sponsored by:	The FreeBSD Foundation
2015-07-21 03:18:53 +00:00
Brad Davis
a54bd5953b Add support for building VirtualBox Vagrant images.
Abstract the build, package and upload to handle building either type.

Approved by:	re (gjb)
2015-07-20 16:27:44 +00:00
Glen Barber
a1912cc902 Do not mount /var/log and /var/tmp as md(4)-backed devices.
Discussed with:	ian
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-07-12 19:17:05 +00:00
Glen Barber
a1c0309edc Revert r285018, r285019, r285076, r285078, r285082.
The true cause of the missing UFS/MSDOSFS labels has been
identified, and only affects stable/10 at the moment.

An request for commit to stable/10 will be pending RE approval
after this commit.

MFC after:	1 day
X-MFC-Note:	never
X-MFC-Never:	r285018, r285019, r285076, r285078, r285082
Sponsored by:	The FreeBSD Foundation
2015-07-04 00:01:41 +00:00
Glen Barber
ef3a5d20dd Test for loader.rc existence before attempting to create
the symlink from loader.rc.sample.

Fix paths relative to the CHROOTDIR.

MFC after:	3 days
X-MFC-With:	r285076, r285078
X-MFC-Before:	10.2-BETA1
Sponsored by:	The FreeBSD Foundation
2015-07-03 06:15:54 +00:00
Glen Barber
174fe0cfce Revert r285018 and r285019, which attempted to resolve
UFS/MSDOSFS label issues on FreeBSD/arm builds, however
the real problem was addressed in r285076, which is due
to two separate issues, unrelated to md(4) stale device
existence.

MFC after:	3 days
X-MFC-With:	r285076
X-MFC-Before:	10.2-BETA1
Sponsored by:	The FreeBSD Foundation
2015-07-03 06:07:21 +00:00
Glen Barber
a6d3c7cdd5 Fix the real problem with missing UFS/MSDOSFS labels on the
FreeBSD/arm builds.  The problem stems from the loader.rc file
not existing, as well as geom_label not being loaded at boot.

For now, add the geom_label_load entry to loader.conf, and
symlink loader.rc.sample to loader.rc, both of which allowed
my BeagleBone Black to boot fine with a UFS label reference in
fstab(5).

MFC after:	3 days
X-MFC-Before:	10.2-BETA1
Sponsored by:	The FreeBSD Foundation
2015-07-03 06:01:37 +00:00
Glen Barber
e8d7333d7d Fix the path to the UFS/MSDOSFS filesystem labels, since they
cannot possibly exist within the chroot(8) before the target
filesystem actually exists.

MFC after:	3 days
X-MFC-With:	r285018
Sponsored by:	The FreeBSD Foundation
2015-07-02 02:33:58 +00:00
Glen Barber
e07ca0423d Implement an evil workaround that prevents UFS/MSDOS labels from being
written to disk with newfs(8) and newfs_msdosfs(8).

When iterating through snapshot builds in serial, it is possible for
a build failure to leave stale md(4) devices behind, in some cases, they
could have a UFS or MSDOS filesystem label assigned.

If the md(4) is not destroyed (or not able to be destroyed, as has
happened recently due to my own fault), the filesystem label that
already exists can interfere with a new md(4) device that is targeted to
have the same label.

This behavior, although admittedly a logic error in the wrapper build
scripts, has caused intermittent reports (in particular with the armv6
builds) of missing UFS/MSDOSFS labels, causing the image to fallback to
the mountroot prompt.  This appears to only happen when the backing
md(4) device is destroyed before the calling umount(8) on the target
mount, after which the UFS/MSDOSFS label persists.

The workaround is this:  If EVERYTHINGISFINE is set to non-empty value,
check for an existing ufs/rootfs and msdosfs/MSDOSBOOT filesystem label
in arm_create_disk(), and rm(1) them if they exist.

The EVERYTHINGISFINE variable is chosen because it is used in exactly
one other place - release/Makefile.mirrors - and there are big scary
warnings at the top of that file as well that it should *not* be used
under normal circumstances.  This should not destroy a build machine
that also uses '/dev/ufs/rootfs' as the UFS label, and I have verified
in extensive local testing that the destroyed label is recreated when
the md(4) is unmounted/mounted, but this really should not be enabled
by anyone.

Having said all that, I absolutely *do* plan MFC this to stable/10 for
the 10.2-RELEASE cycle, as so far, I have only observed this behavior
on stable/10, but this is a temporary solution until I can unravel all
of the failure paths to properly trap them.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-07-02 02:13:20 +00:00
Brad Davis
41f7db3a4f Add initial support for building Vagrant images for VMWare. Next steps will be adding Virtualbox support and uploading to Hashicorp Atlas for others to consume.
Approved by:	re (gjb)
2015-06-27 20:39:13 +00:00
Glen Barber
eb64cd58f5 Add initial support for automatically uploading GCE virtual
machine images to the Google Compute Engine platform.

By default, gcutil/gsutil requires an Oauth2 login generated
from a URL that must be opened in a browser, a verification
code copied back to the terminal from which it was invoked,
etc., etc., making it near impossible for automation.

I've hacked together an evil solution to work around this,
so unless GCE_LOGIN_SKIP is set to a non-empty value, this
Makefile will not do anything useful.

As a result of this commit, remove the gce-package.sh script
that was never, nor will ever be, used.

MFC after:	3 days
X-MFC-Note:	(hopefully)
Sponsored by:	The FreeBSD Foundation
2015-06-27 01:29:59 +00:00
Glen Barber
67076e4d58 Ensure ASSUME_ALWAYS_YES is set when deleting pkg(8) from the
DESTDIR, otherwise it can get stuck in ttyin.

This means no 10-STABLE EC2 images this week.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-24 21:30:35 +00:00
Glen Barber
798e3ec502 Chase r284656 and r284658:
Remove the Azure-local vm_extra_create_disk(), since we no longer
need qemu-img to convert the final VHD image to an Azure-compatible
format.

Although the waagent utility is installed from ports, create the
symlink to /usr/sbin, pending investigation on where this is
hard-coded, so it can be reported upstream.  In the meantime, this
is good enough.

MFC after:	3 days
X-MFC-Needs:	r284269, r284270, r284271, r284655,
		r284656, r284657, r284658, r284659
X-MFC-Note:	Required for 10.2-RELEASE, marcel@ has
		implicit approval for the required changes
Sponsored by:	The FreeBSD Foundation
2015-06-21 04:34:57 +00:00
Glen Barber
99b858f537 Fix path to waagent.
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-19 23:03:05 +00:00
Glen Barber
717a45fcae Work around a potential bug in pw(8) when '-m' is specified.
According to the manual page, '-m' should create the user home
directory, however rigorous testing suggests it does not, and
it is unclear if this is an implementation or expectation issue.

Sponsored by:	The FreeBSD Foundation
2015-06-05 02:22:04 +00:00
Glen Barber
0bf3ddb7a7 Make sure /home exists within the DESTDIR before creating the
'freebsd' user account.

Sponsored by:	The FreeBSD Foundation
2015-06-04 23:11:24 +00:00
Glen Barber
b07d911f39 Use the new pw(8) '-R' option to create the users/groups.
MFC after:	1 week
X-MFC-after:	r283961
Sponsored by:	The FreeBSD Foundation
2015-06-03 19:19:25 +00:00
Glen Barber
92284ce807 Pass the '-b' argument to pw(8) so the '/home' directory is created
in the correct place.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-02 15:02:30 +00:00
Glen Barber
915f44d03b In arm_create_disk(), disable soft updates journaling.
Disabling soft updates journaling appears to resolve issues
with kernel panics, and may also be generally bad to have
enabled for SD cards.

Requested by:	ian
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-01 21:10:47 +00:00
Glen Barber
f3ca4d9170 Re-enable arm_create_user(), and pass the '-V DESTDIR/etc' to
pw(8) to set the correct /etc directory for the user/group
files.

Provided by:	ian (thanks!)
MFC after:	3 days
X-MFC-with:	r283894
Sponsored by:	The FreeBSD Foundation
2015-06-01 20:59:18 +00:00
Glen Barber
b506d010c2 Disable arm_create_user(). As-is, this does not create the
user in the userland for the target image, but creates the
user in the build chroot.

Before this is re-enabled, I want to figure out a clean way
to do this without requiring the overhead of third-party
utilities (such as qemu).

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-06-01 20:26:38 +00:00
Glen Barber
ccbf25027e Fix virtual machine disk format creating by passing VMFORMAT
to mkimg(1)

PR:		200068
Submitted by:	Jeremy Norris
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-05-11 14:53:07 +00:00
Glen Barber
88feae84be Merge ^/projects/release-arm-redux into ^/head.
Of note:

- This commit adds native FreeBSD/arm release build support without
  requiring out-of-tree utilities.

- Part of this merge removes the WANDBOARD-{SOLO,DUAL,QUAD} kernel
  configuration files, for which the IMX6 kernel configuration file
  should be used instead.

- The resulting images have a 'freebsd' user (password 'freebsd'),
  to allow ssh(1) access when console access is not available (VGA
  or serial).  The default 'root' user password is set to 'root'.

- The /etc/ttys file for arm images now enable both ttyv0 and ttyu0
  by default.

Help from:	many (boot testing, feedback, etc.)
Sponsored by:	The FreeBSD Foundation
2015-05-09 21:08:12 +00:00
Glen Barber
fef41f600c Fix path to /boot/msdos for the target image.
Sponsored by:	The FreeBSD Foundation
2015-05-08 10:38:32 +00:00
Glen Barber
869ef34722 Reorder / and /boot/msdos mount to avoid the following error:
Mounting local file systems:
mount: /boot/msdos: No such file or directory

Sponsored by:	The FreeBSD Foundation
2015-05-07 23:46:33 +00:00
Glen Barber
efc6ab44f2 Create the /boot/msdos directory so the FAT partition is mounted
properly.

Noticed by:	sbruno
Sponsored by:	The FreeBSD Foundation
2015-05-07 20:25:17 +00:00
Glen Barber
60bc17f490 Enable the growfs rc(8) script.
Sponsored by:	The FreeBSD Foundation
2015-05-07 18:49:43 +00:00
Glen Barber
04b7fd007b In arm_create_disk(), chroot(8) to the build environment
is necessary.

In arm_install_base(), chroot(8) when installing world
and kernel.  Fix paths for fstab(5) and rc.conf(5).

Sponsored by:	The FreeBSD Foundation
2015-05-07 17:35:48 +00:00
Glen Barber
befaadd56c In release.sh, create the /dev directory within the chroot
before attempting to mount(8) devfs.  Also, create the
.OBJDIR for the 'release' target, so files end up in the
correct location.

In tools/arm.subr, fix the target device when creating the
gpart partition scheme.

Sponsored by:	The FreeBSD Foundation
2015-05-07 00:27:28 +00:00
Glen Barber
4ad8b75d53 Add tools/arm.subr to contain common subroutines used for
building arm images.  This is similar to tools/vmimage.subr
used for building virtual machine disk images.  By default,
only arm_create_disk() and arm_install_base() contain real
functionality here, and arm_install_uboot() must be overridden
in the arm/KERNEL.conf file.

In release.sh, make create_arm_armv6_build_release() do
something now.

In arm/BEAGLEBONE.conf, set IMAGE_SIZE, PART_SCHEME, FAT_SIZE,
FAT_TYPE, and MD_ARGS, as well as make arm_install_uboot()
functional.

Parts of this were taken from disecting a previous BEAGLEBONE
image, and other parts obtained from Crochet sources.

Sponsored by:	The FreeBSD Foundation
2015-05-06 19:58:12 +00:00
Glen Barber
fea236302d Remove crochet configuration files and tools/arm/ directory.
Sponsored by:	The FreeBSD Foundation
2015-05-06 15:59:29 +00:00
Glen Barber
ace79aa796 Fix building BEAGLEBONE images with Crochet using the
sysutils/u-boot-beaglebone port:

 - In arm/BEAGLEBONE.conf, set EMBEDDEDPORTS to the
   sysutils/u-boot-beaglebone port.
 - In arm/release.sh, remove BEAGLEBONE from setting WANT_UBOOT
 - In tools/arm/crochet-BEAGLEBONE.conf, override the
   beaglebone_check_uboot(), and set BEAGLEBONE_UBOOT to
   /tmp/external/u-boot-beaglebone, and create symlinks to the
   u-boot files in /usr/local/share/u-boot-beaglebone and the
   uEnv.txt file in crochet/board/Beaglebone/files.

Sponsored by:	The FreeBSD Foundation
2015-05-05 21:08:50 +00:00
Glen Barber
20a67b0520 Add initial support for building RPI2 images.
In release.sh, allow overriding buildenv_setup() before
the handoff to arm/release.sh.

Copy arm/RPI-B.conf -> arm/RPI2.conf, set UBOOT_PORT and
the correct KERNEL, and add the buildenv_setup() override
to install the sysutils/u-boot-rpi2 port/package.

Copy tools/arm/crochet-RPI-B.conf -> tools/arm/crochet-RPI2.conf,
and set the correct entries for the RaspberryPi2 board.

Thanks to:	loos@
Sponsored by:	The FreeBSD Foundation
2015-05-05 16:28:04 +00:00
Glen Barber
26e4122a80 Fix a comment.
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-04-27 19:49:50 +00:00
Glen Barber
08a75d1f0e Create the arm64/aarch64 VM disk image as MBR instead of
GPT scheme.  UEFI needs to know the unique partition GUID
with GPT, which changes each time.  Specifically, the QEMU
EFI BIOS file has this hard-coded.[1]

Since the GPT labels are now unavailable, unconditionally
label the root filesystem as 'rootfs' with newfs(8), since
it does not hurt anything anywhere else.  For the arm64 case,
'/' is mounted from /dev/ufs/rootfs; for all other VM images,
'/' is mounted from /dev/gpt/rootfs.

Unfortunately, since the /dev/gpt/swapfs label is also lost,
set NOSWAP=1 for the arm64/aarch64 images.  This is temporary,
until I figure out a scalable solution to this.  But, a certain
piece of softare was written "very fast", and ended up living
for 15 years.  We can deal with this for a week or so.

Information from:	andrew, emaste [1]
Sponsored by:	The FreeBSD Foundation
2015-04-23 01:04:14 +00:00
Glen Barber
bf93ac7884 Revert r281809, which did more harm than good, and apply
a more proper fix when attempting to locate the /boot
files.

Sponsored by:	The FreeBSD Foundation
2015-04-21 14:48:38 +00:00
Glen Barber
fa56961588 Simplify variable expansion in attempt to fix the vm-image build.
Sponsored by:	The FreeBSD Foundation
2015-04-21 02:49:44 +00:00
Glen Barber
6a3afe6924 Add support for building arm64/aarch64 virtual machine images.
Sponsored by:	The FreeBSD Foundation
2015-04-20 20:06:25 +00:00
Glen Barber
4b8175ee8f When building VM disk images, vm_copy_base() uses tar(1) to
copy the userland from one md(4)-mounted filesystem to a clean
filesystem to prevent remnants of files that were added and
removed from resulting in an unclean filesystem.  When newfs(8)
creates the first filesystem with journaled soft-updates enabled,
the /.sujournal file in the new filesystem cannot be overwritten
by the /.sujournal in the original filesystem.

To avoid this particular error case, do not enable journaled
soft-updates when creating the md(4)-backed filesystems, and
instead use tunefs(8) to enable journaled soft-updates after
the new filesystem is populated in vm_copy_base().

While here, fix a long standing bug where the build environment
/boot files were used by mkimg(1) when creating the VM disk
images by using the files in .OBJDIR.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-04-20 19:54:54 +00:00
Colin Percival
450a69e70c It would help if I committed the right patch... 2015-03-31 04:35:35 +00:00
Colin Percival
6e8e6c9951 Partially revert r278118 now that the required logic for deciding whether
freebsd-update can be useful has moved into the firstboot_freebsd_update
script.
2015-03-31 01:44:29 +00:00
Colin Percival
f65557efc4 Add bits for building EC2 disk images. Make logic for uploading these
to create EC2 AMIs will come in a later commit.
2015-03-31 01:28:33 +00:00
Glen Barber
e618222a18 Sigh. s/AutoSize/Growfs/ following upstream commit r761.
MFH:		3 days
Sponsored by:	The FreeBSD Foundation
2015-03-30 19:08:37 +00:00
Colin Percival
dcdebabdea Improve check for whether ${DESTDIR}/dev is mounted.
Submitted by:	gcooper
2015-03-30 09:08:24 +00:00
Colin Percival
963412283a Clean up filesystem unmounting in vmimage builds:
* Remove vm_umount_base function which is currently unused.
* Add umount_loop function which loops attempting to unmount one filesystem.
* Replace calls to umount with calls to umount_loop.
* Don't attempt to unmount ${DESTDIR}/dev if it isn't mounted.

The looping is necessary because sometimes umount fails due to filesystems
being busy.  The most common cause of such busyness is periodic(8) jobs
running `find / ...`.

Reviewed by:	gjb
2015-03-30 08:33:19 +00:00
Colin Percival
25c1155771 When creating VM images, copy the contents of the created filesystem into
a new filesystem before packaging it into a disk image.  This prevents
"remnants" of deleted files from showing up in the VM images, and reduces
their compressed size (by about 10% for the cloudware images) as a result.

Looks good to:	gjb
2015-03-20 19:40:19 +00:00
Glen Barber
dc1c45001a Add a CLOUDPACKAGE variable to contain a list of targets for
post-install packaging cloud provider images.

Add a 'gce-package.sh' script to generate the final output
image ready for upload to the GCE platform.  Right now, this
is the only image that has a specific output format (GNU-tar),
and this implementation is expected to be temporary.

This is not directly connected to the other release targets.

MFC after:	1 week
X-MFC-with:	r279249, r279250
Sponsored by:	The FreeBSD Foundation
2015-02-24 22:27:02 +00:00
Glen Barber
3e6090ebb5 Use ifconfig_DEFAULT="" instead of hard-coding vtnet(4)
MFC after:	2 weeks
X-MFC-with:	r277458, r277536, r277606, r277609,
		r277836, r278118
Sponsored by:	The FreeBSD Foundation
2015-02-03 00:10:48 +00:00