Commit Graph

457 Commits

Author SHA1 Message Date
Bryan Drewery
30385347cf Rework r254951 to not reach into private OBJDIR areas.
The original change was dealing with the build wanting to run a newer
install(1) that was not yet installed.  The solution to look into the private
legacy directory of the existing build conflicts with 2 upcoming features: a
changed OBJDIR format, and splitting the host tools into arch-dependent and
arch-independent directories.  Rather than hardcoding and changing the paths in
this script, just let kernel-toolchain do the work, while disabling much of the
meat.  With -j15 this finishes in 25 seconds for me and 117 seconds with -j1.
All that is really needed is bootstrap-tools, but the system is not currently
written in a way that all previous dependent steps will have ran.  The previous
steps, such as _worldtmp, are being reworked and renamed and so cannot be
relied upon to be right.

Sponsored by:	Dell EMC Isilon
2017-10-29 01:21:09 +00:00
Glen Barber
5567474c4a Fix Vagrant image upload after recent API changes.
- Update ATLAS_UPLOAD_URL to avoid various regular expressions
  from failing to match due to redirections.
- Use ATLAS_UPLOAD_URL throughout the script.
- Adjust several regular expression patterns.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-07-03 14:31:38 +00:00
Glen Barber
fefd043963 Allow PORTSDIR to be overridden in the pkg-stage target.
Submitted by:	Johannes Jost Meixner (original)
PR:		218393
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-05-22 14:21:45 +00:00
Glen Barber
4d8276988e Update release/scripts/atlas-upload.sh to account for API changes
made recently by Atlas Hashicorp.  The data returned from GET and
POST requests has changed, which caused a number of regex patterns
to fail to be properly identified as 'success' or 'failure', which
ended up in upload/publish failures.

Tested with:	12-CURRENT
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2017-05-11 16:37:29 +00:00
Ed Maste
868f280a54 Add sysutils/tmux to the DVD package set
The zsh + tmux combination is commonly wanted.

Approved by:	gjb
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
2017-04-24 16:37:25 +00:00
Matthew Seaman
aeb76c0bb7 Revert r309339, thus re-instating r309314
The original problem with conflicting definitions of ${PKG_CMD} was
solved by r427523 in ports (see https://reviews.freebsd.org/D8677), so
this should be safe now.

Reviewed by:	gjb
Approved by:	gjb
Differential Revision:	https://reviews.freebsd.org/D8120
2016-12-16 23:05:47 +00:00
Glen Barber
ebc7f2b8de Revert r309314, which breaks installing ports.
Requested by:	antoine
Differential Revision:	https://reviews.freebsd.org/D8120 (related)
Sponsored by:	The FreeBSD Foundation
2016-11-30 22:00:25 +00:00
Matthew Seaman
91c915facf Allow a user-overridable setting 'PKG_CMD' to control the command used
to create a repo during 'make packages'

This would have been useful for a situation I found myself in where
pkg(8) had been upgraded to a version that wanted the FBSD_1.5 ABI
version but libc.so.7 had not been upgraded, and only provided
FBSD_1.4. I found I needed to update libc in order to run pkg, and I
also needed to use pkg to update libc... Which is why pkg-static
exists, but there's currently no way to tell the build system to use
pkg-static instead of pkg.

This creates a variable PKG_CMD, default value 'pkg', that can be
overridden from the command line.

Reviewed by:	gjb
Approved by:	gjb
Differential Revision:	https://reviews.freebsd.org/D8120
2016-11-30 07:16:29 +00:00
Xin LI
a29af14de4 Apply mergemaster r255428:
Pass -n (do not emit comments) when saving mtree information for future
mergemaster(8) runs.

MFC after:	3 days
2016-06-08 06:33:55 +00:00
Glen Barber
f8b0d3730f Replace linux-f10 packages with updated linux-c6.
PR: 		208280
Submitted by: 	Johannes Meixner
Sponsored by:	The FreeBSD Foundation
2016-05-27 18:20:07 +00:00
Glen Barber
b0bdd78348 Avoid adding the '-release' suffix to non-debug kernel packages.
Sponsored by:	The FreeBSD Foundation
2016-03-08 19:35:30 +00:00
Glen Barber
44d81980ed Prefix lib32, profile, debug, and development packages
with 'runtime-' for consistency with other packages.

Sponsored by:	The FreeBSD Foundation
2016-03-08 18:31:26 +00:00
Glen Barber
3a3f435f98 Include a 'package-pkg' target, intended for use for
architectures we do not provide upstream pkg(8) packages.

This is not tied to anything as-is, and likely will break
your system if used (based on experience with testing with
powerpc).

There is an overwhelming amount of evil happening here,
so until the issues are fixed, it will not be tied into the
'packages' target.

Sponsored by:	The FreeBSD Foundation
2016-02-24 03:05:05 +00:00
Glen Barber
750e7eb5dc Fix naming inconsistency with lib32 debug packages.
Sponsored by:	The FreeBSD Foundation
2016-02-01 18:50:52 +00:00
Glen Barber
3d98f473dc Add 'stagekernel' target, which invokes 'distributekernel'
with -DNO_ROOT to create the METALOG mtree(8) file.

Separate the default STAGEDIR for world (WSTAGEDIR) and kernel
(KSTAGEDIR).

Fix the 'create-kernel-packages' target to work properly.

Evaluate if 'kernel' is set when invoking mtree-to-plist.awk,
which splits the kernel and kernel.debug into separate plist
files.

Fix METALOG creation when building/packaging multiple kernels.

Sponsored by:	The FreeBSD Foundation
2016-01-27 21:10:03 +00:00
Glen Barber
1a5c9277ca MFH
Sponsored by:	The FreeBSD Foundation
2016-01-05 21:12:49 +00:00
Glen Barber
8834318685 Merge ^/projects/release-install-debug:
- Rework MANIFEST generation and parsing via bsdinstall(8).
- Allow selecting debugging distribution sets during install.
- Rework bsdinstall(8) to fetch remote debug distribution sets
  when they are not available on the local install medium.
- Allow selecting additional non-GENERIC kernels during install.
  At present, GENERIC is still required, and installed by default.

Tested with:	head@r293203
Sponsored by:	The FreeBSD Foundation
2016-01-05 21:05:17 +00:00
Glen Barber
b626f5a73a MFH r289384-r293170
Sponsored by:	The FreeBSD Foundation
2016-01-04 19:19:48 +00:00
Simon J. Gerraty
3fc9660791 Ensure TEMPROOT is unique, to guard against multiple builds on same machine.
Differential Revision:	D3002
Reviewed by:	gjb
2015-11-08 23:46:05 +00:00
Baptiste Daroussin
9be27fdc00 Merge from head 2015-10-09 22:45:54 +00:00
Craig Rodrigues
c151731fbf Use print as a function, not operator. 2015-10-08 03:28:15 +00:00
Baptiste Daroussin
b5ff185e19 Merge from head 2015-09-12 11:41:31 +00:00
Brad Davis
a54bd5953b Add support for building VirtualBox Vagrant images.
Abstract the build, package and upload to handle building either type.

Approved by:	re (gjb)
2015-07-20 16:27:44 +00:00
Brad Davis
f19a742ff1 Add Support for uploading Vagrant images to Hashicorp Atlas.
Approved by:	re (gjb)
2015-06-29 21:26:41 +00:00
Baptiste Daroussin
7757a1b4dc Merge from head 2015-05-03 19:30:11 +00:00
Glen Barber
08a75d1f0e Create the arm64/aarch64 VM disk image as MBR instead of
GPT scheme.  UEFI needs to know the unique partition GUID
with GPT, which changes each time.  Specifically, the QEMU
EFI BIOS file has this hard-coded.[1]

Since the GPT labels are now unavailable, unconditionally
label the root filesystem as 'rootfs' with newfs(8), since
it does not hurt anything anywhere else.  For the arm64 case,
'/' is mounted from /dev/ufs/rootfs; for all other VM images,
'/' is mounted from /dev/gpt/rootfs.

Unfortunately, since the /dev/gpt/swapfs label is also lost,
set NOSWAP=1 for the arm64/aarch64 images.  This is temporary,
until I figure out a scalable solution to this.  But, a certain
piece of softare was written "very fast", and ended up living
for 15 years.  We can deal with this for a week or so.

Information from:	andrew, emaste [1]
Sponsored by:	The FreeBSD Foundation
2015-04-23 01:04:14 +00:00
Baptiste Daroussin
4bf53d0b46 Merge from HEAD 2015-04-03 23:23:09 +00:00
Colin Percival
25c1155771 When creating VM images, copy the contents of the created filesystem into
a new filesystem before packaging it into a disk image.  This prevents
"remnants" of deleted files from showing up in the VM images, and reduces
their compressed size (by about 10% for the cloudware images) as a result.

Looks good to:	gjb
2015-03-20 19:40:19 +00:00
Baptiste Daroussin
6b710cc6fe development and profile are special tags and should always be the last component
of the name if present
2015-03-18 11:27:11 +00:00
Baptiste Daroussin
0a48fbc082 Support a config tag to flag a file that should be merge on upgrade 2015-03-05 16:20:53 +00:00
Baptiste Daroussin
1e53d87c00 Convert tagged directories in mtree into their plist form 2015-03-05 11:43:12 +00:00
Baptiste Daroussin
f1d1786e24 Simplify and clean awk script
Suggested by:	dteske
2015-03-05 00:18:32 +00:00
Baptiste Daroussin
eed8bc1862 If the package is "runtime-something" rename into "something" 2015-03-04 23:53:45 +00:00
Baptiste Daroussin
f4da3de647 Improve mtree to plist converter
1/ Analyse the fields from mtree lines to extract:
 - owner,
 - group,
 - mode,
 - flags,
 - tags,
2/ Ignore non tagged entries
3/ For now concat tags to form a futur package name
2015-03-04 23:45:39 +00:00
Baptiste Daroussin
d8fee543ec Merge from HEAD 2015-02-12 15:34:16 +00:00
Colin Percival
11d9aa6707 Step 1 of eliminating the "games" distribution: Move binaries to /usr/bin;
update paths; and include everything in the "base" distribution.

The "games" distribution being optional made sense when there were more
games and we had small disks; but the "games-like" games were moved into
the ports tree a dozen years ago and the remaining "utility-like" games
occupy less than 0.001% of my laptop's small hard drive.  Meanwhile every
new user is confronted by the question "do you want games installed" when
they they try to install FreeBSD.

The next steps will be:

2. Removing punch card (bcd, ppt), phase-of-moon (pom), clock (grdc), and
caesar cipher (caesar, rot13) utilities.  I intend to keep fortune, factor,
morse, number, primes, and random, since there is evidence that those are
still being used.

3. Merging src/games into src/usr.bin.

This change will not be MFCed.

Reviewed by:	jmg
Discussed at:	EuroBSDCon
Approved by:	gjb (release-affecting changes)
2015-02-12 05:35:00 +00:00
Glen Barber
04bd90c634 Bump copyright, forgotten in r278502.
MFC after:	1 week
X-MFC-with:	r277458, r277536, r277606, r277609,
		r277836, r278118, r278119, r278206,
		r278502
Sponsored by:	The FreeBSD Foundation
2015-02-10 12:38:39 +00:00
Glen Barber
e90de53410 In scripts/mk-vmimage.sh, prevent incorrect usage() by
defaulting VMCONFIG to /dev/null, and additionally
ensuring VMCONFIG is not a character device before it
is sourced.

While here, be sure to exit if usage() is called.

This should effectively be no-op, but the usage() output
was discovered while investigating a larger issue.

MFC after:	1 week
X-MFC-with:	r277458, r277536, r277606, r277609,
		r277836, r278118, r278119, r278206
Sponsored by:	The FreeBSD Foundation
2015-02-10 12:04:38 +00:00
Baptiste Daroussin
24a110a1ec Add a awk script to convert the metalog output into plist
Splitting according to tags
2015-02-08 18:06:36 +00:00
Glen Barber
5a5f93de64 Fix a typo in a comment.
MFC after:	1 week
X-MFC-with:	r277837
Sponsored by:	The FreeBSD Foundation
2015-01-28 17:39:57 +00:00
Glen Barber
429c554fd6 Provide a description for the 'tests.txz' distribution
in the MANIFEST file.

Turn off the test suite installation by default.

Tested with:	head@r277834
MFC after:	1 week
Sponsored by:	The FreeBSD Foundation
2015-01-28 17:34:27 +00:00
Glen Barber
2d4ff62f00 Merge the following revisions from ^/projects/release-vmimage:
r273823-r273826, r273833, r273836, r273944, r274069-r274071,
 r274134, r274211, r274280-r274285, r274287-r274288, r274292,
 r274296-r274297, r274356, r274533, r274725, r274726, r274729,
 r274734, r274771, r274945-r274946, r277180, r277183-r277184,
 r277186-r277187, r277250-r277253, r277263-r277264, r277383-r277384,
 r277393-r277395, r277438-r277439, r277447, r277455:

 r273823:
  Move virtual machine / cloud provider targets and
  options from release/Makefile to their own Makefile.

 r273824:
  Add glue to allow enabling building cloud provider VM images
  by default.

  When WITH_CLOUDWARE is not empty, add CLOUDTARGETS to the
  release/Makefile 'release' target.

 r273825:
  Avoid hard-coding the Azure image file format.  While here,
  avoid using OSRELEASE for the output file name.

 r273826:
  Remove a few vestiges of passing an exit code to panic().

 r273833:
  Initial commit providing a mechanism to create openstack images
  as part of the release build.

 r273836:
  Fix output file name for openstack images.  No further conversion
  is necessary for this VM file target, so there is no need to append
  the '.raw' suffix here.

 r273944:
  Uncomment the cloudinit rc.conf(5) line.

 r274069:
  Add line continuation so OPENSTACKCONF is actually included in the env(1).

 r274070:
  Add a 'vm-cloudware' target, used to drive all targets in CLOUDTARGETS.

 r274071:
  Add examples for WITH_CLOUDWARE to release.conf.sample.
  Add WITH_CLOUDWARE evaluation to RELEASE_RMAKEFLAGS.

 r274134:
  Initial rewrite to consolidate VM image build scripts into one.

 r274211:
  Add write_partition_layout() used to populate the final image.

  Fix duplicated mkimg(1) call in vm_create_disk().

  Add primitive (untested) PowerPC/PowerPC64 VM image support.

  Note: As it is currently written, the /boot/pmbr and
  /boot/{gptboot,boot1.hfs} use the build host and not the target
  build.  Fixing this is likely going to be a hack in itself.

 r274280:
  Return if vm_create_disk() is unsuccessful.

 r274281:
  Add CLEANFILES entry for VM targets

 r274282:
  Add vm_extra_pre_umount() prototype to vmimage.subr.

 r274283:
  Fix DESTDIR for installworld, and make sure it is created before use.

 r274284:
  Move usage() from vmimage.subr to mk-vmimage.sh, in case vmimage.subr
  has not been sourced.

 r274285:
  Spell 'OPTARG' correctly.  Actually call vm_create_base().

 r274287:
  Fix line continuation in write_partition_layout().
  Remove variable test that is no longer needed.

 r274288:
  Fix scheme flag to mkimg(1).

 r274292:
  mount(8) and umount(8) devfs(5) as needed.

 r274296:
  Change path for mk-vmimage.sh from ${TARGET}/ to scripts/ now that
  it is consolidated into one file.

  Fix paths for the base image and output disk image files.

 r274297:
  Call cleanup() after everything is done.

 r274356:
  Remove a stray directory from CLEANFILES.

 r274533:
  Set the boot partition type to 'apple-boot' for powerpc.

 r274725:
  In vm_install_base(), copy the host resolv.conf into
  the build chroot before attempting to do anything that
  requires working DNS (i.e., pkg bootstrap).

  In vm_extra_pre_umount(), remove the resolv.conf before
  the disk image is unmounted from the backing md(4).

 r274726 (cperciva):
  Silence errors when umounting the chroot's /dev, since it
  probably doesn't exist when we're running this.

  Unmount filesystems before attempting to destroy the md which
  holds them.

 r274729 (cperciva):
  Unmount filesystem and destroy md before we read the vnode from
  disk and package it into a disk image.  Otherwise we end up
  packaging an unclean filesystem.

 r274734 (cperciva):
  Merge duplicative vm-CLOUDTYPE targets before additional duplication
  gets added by the impending arrival of ec2 and gcloud.

 r274771 (cperciva):
  Add NOSWAP option which can be set by a vmimage.conf file to specify
  that no swap space should be created in the image.  This will be used
  by EC2 builds, since FreeBSD/EC2 allocates swap space on "ephemeral"
  disks which are physically attached to the Xen host node.

 r274945:
  In vm_extra_install_packages(), only bootstrap pkg(8) if
  VM_EXTRA_PACKAGES is empty.

  In vm_extra_pre_umount(), cleanup downloaded packages if pkg(8) was
  bootstrapped earlier.

 r274946:
  Fix indentation nit.

 r277180:
  In vm_extra_install_base(), do not install waagent in the openstack
  image, because it is not used.  This appears to be a copy mistake.

  Remove vm_extra_install_base() from the openstack.conf entirely,
  since it does not need to be overridden.

 r277183:
  Enable the textmode console by default for VM images, since there is
  no way to tell if the environment will be able to use the
  graphics-mode console.

 r277184:
  Enable password-less sudo for openstack images.

 r277186:
  Update the VM_EXTRA_PACKAGES list for the openstack images.

  The documentation suggests doing a "just fetch this and run it"-style
  bootstrap, from which the list of dependencies was obtained (in
  github, at: pellaeon/bsd-cloudinit-installer)

  There is one Python dependency unmet, oslo.config, which is not in
  the Ports Collection.

 r277187:
  Add a comment to note that setting hw.vga.textmode=1 is temporary.

 r277250:
  Remove vm_extra_install_base() for the Azure image, now that the
  waagent exists in the ports tree.

  Add sysutils/azure-agent to the VM_EXTRA_PACKAGES list.

  In vm_extra_pre_umount(), remove the explicit pkg(8) install
  list, as dependencies are resolved by sysutils/azure-agent.

 r277251:
  Add a 'list-cloudware' target to print the list of supported CLOUDWARE
  values and a description.

  Add the AZURE_DESC and OPENSTACK_DESC descriptions.

 r277252:
  Update release(7)

 r277253:
  Add 'list-vmtargets' target, which produces a list of all supported
  VM and cloud provider images.

  Add VHD_DESC, VMDK_DESC, QCOW2_DESC, RAW_DESC image descriptions.

  Format the output to make a bit more readable.

  Update release(7) to document the list-vmtargets target.

 r277263:
  Add initial support for the GCE (Google Compute Engine) cloud hosting
  provider image.

 r277264:
  Style and line length cleanup.

 r277383:
  Remove the console setting from rc.conf(5), which is not used there.
  While here, set console to include vidconsole in the loader.conf(5).

 r277384:
  Fix an indentation nit.
  No functional changes.

 r277393:
  Remove the pkg-clean(8) call from vm_extra_pre_umount() since the
  function is often overridden.

  Add vm_extra_pkg_rmcache() to call pkg-clean(8) to avoid duplicated
  code.

 r277394:
  Move resolv.conf(5) removal back to vm_extra_pre_umount() where it
  belongs.

  The GCE image needs resolv.conf(5) to exist (created as part of the
  image setup), so it cannot be removed.

 r277395:
  Comment the line that configures ttys(5) to 'off', which makes it
  impossible to test that the image boots.

  Add a note explaining why the line is commented, and not (yet) removed
  entirely.

 r277438:
  Move the 'install' bits that are specific to virtual machine images
  from the Makefile to Makefile.vm.

  Rename the 'install' target to 'release-install', and add a new
  'vm-install' target.

  Add a new 'install' target that invokes the new targets.

 r277439:
  Add WITH_CLOUDWARE to the list of make(1) variables for the release
  build.

 r277447:
  Remove hw.vga.textmode=1 from the VM image loader.conf, which was
  included during test builds and not intended to be included when
  merging this project branch back to head.

 r277455:
  Remove mk-azure.sh, which is no longer needed.

MFC after:	1 month
X-MFC-To:	stable/10 (requires mkimg(1))
Help from:	cperciva, swills
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
2015-01-20 23:56:04 +00:00
Glen Barber
8f0ea33f2b Reintegrate head revisions r273096-r277147
Sponsored by:	The FreeBSD Foundation
2015-01-13 21:29:24 +00:00
Glen Barber
2303632301 Switch to x11/gnome3 now that x11/gnome2 no
longer exists in the ports tree.

MFC after:	3 days
X-MFC-With:	r276820, r276822, r276827
Sponsored by:	The FreeBSD Foundation
2015-01-08 15:50:10 +00:00
Glen Barber
0669938973 Ensure the ports directories exist for the list of
packages intended to be included on the DVD, and
remove any nonexistent ports from the final list.

Print the list of missing paths, and ensure that
DVD_PACKAGES is non-zero length (which should never
happen).

MFC after:	3 days
X-MFC-With:	r276820, r276822
Sponsored by:	The FreeBSD Foundation
2015-01-08 15:42:10 +00:00
Glen Barber
6b7ca552c3 Ensure /var/run/ld-elf.so.hints exists before building
pkg(8).

MFC after:	3 days
X-MFC-With:	r276820
Sponsored by:	The FreeBSD Foundation
2015-01-08 14:39:53 +00:00
Glen Barber
ef6db57978 Print informational output when NOPORTS is set,
which would otherwise cause pkg-stage.sh to
silently exit.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-01-08 14:29:03 +00:00
Glen Barber
83373daa35 Properly change directories before creating the
PKG_ALTABI -> PKG_ABI symlink.

MFC after:	3 days
X-MFC-With:	r276765
Sponsored by:	The FreeBSD Foundation
2015-01-06 20:46:29 +00:00
Glen Barber
8f85f8e66e Update pkg-stage.sh to be compatible with pkg-1.4.x.
In 1.3.x and earlier, ABI format is, for example,
freebsd:11:x86:64.

In 1.4.x, ABI format is FreeBSD:11:amd64, and a new
configuration entry, ALTABI, is: freebsd:11:x86:64.

Export PKG_ABI and PKG_ALTABI accordingly, and if
PKG_ALTABI is set, create a symlink within the dvd
PKG_CACHEDIR so both new and old ABI directories
exist.

MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
2015-01-06 20:42:50 +00:00
Glen Barber
4c16f1b111 Adjust the 'Relnotes:' search to look for any non-empty
string following the tag.

Sponsored by:	The FreeBSD Foundation
2014-12-08 15:45:07 +00:00