Commit Graph

1781 Commits

Author SHA1 Message Date
Bryan Drewery
36269b8231 Fix native-xtools build
Fixes		https://github.com/freebsd/poudriere/issues/894
Fixes:		d0c737e18 ("Makefile: Fix MAKEOBJDIRPREFIX command-line")
X-MFC-With:	d0c737e18
2021-08-03 08:22:14 -07:00
Alex Richardson
2de949cf85 Remove mkcsmapper_static and mkesdb_static from build-tools
Build them as part of the bootstrap-tools phase instead.

Reviewed by:	emaste (no objections)
Differential Revision: https://reviews.freebsd.org/D28181
2021-08-02 14:33:25 +01:00
Alex Richardson
7bc797e3f3 Add build system support for ASAN+UBSAN instrumentation
This adds two new options WITH_ASAN/WITH_UBSAN that can be set to
enable instrumentation of all binaries with AddressSanitizer and/or
UndefinedBehaviourSanitizer. This current patch is almost sufficient
to get a complete buildworld with sanitizer instrumentation but in
order to actually build and boot a system it depends on a few more
follow-up commits.

Reviewed By:	brooks, kib, markj
Differential Revision: https://reviews.freebsd.org/D31043
2021-08-02 14:33:24 +01:00
Wolfram Schneider
bbd16236e9 add the time(1) command to the list of install tools
Reported by:	dhw
Approved by:	dhw
Differential Revision:	https://reviews.freebsd.org/D31373
2021-08-01 13:25:00 +00:00
Wolfram Schneider
99feb137f5 `make buildworld' with time logging for each stage
PR:	 	257141
Reviewed by:	sjg,emaste
Approved by:	emaste
Differential Revision:	https://reviews.freebsd.org/D31154
2021-08-01 09:10:49 +00:00
Ed Maste
4dbac528db pkgbase: improve pkg --version parsing
In some cases `pkg --version` might produce unexpected or additional
output.  Use a regex /^[0-9.]+$/ to match only the line containing the
version number.

Reported by:	Michael Butler on freebsd-current@
Fixes:		4e224e4be7 ("pkgbase: accommodate pkg < 1.17")
Sponsored by:	The FreeBSD Foundation
2021-07-28 17:14:33 -04:00
Ed Maste
4e224e4be7 pkgbase: accommodate pkg < 1.17
6cafdee71d adapted the pkgbase build for 1.17, but broke Cirrus-CI's
use of PKG_FORMAT=tar (the quarterly package set still has pkg 1.16).
Because of this I disabled the pkgbase build and test in 2bfba2a04b.

Now, check `pkg --version` and use the old logic for < 1.17.

To be reverted once we no longer encounter pkg 1.16 in Cirrus-CI (i.e.,
via GCP cloud images) to avoid keeping this extra complexity around.

PR:		257422
Reviewed by:	manu
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D31324
2021-07-27 14:50:06 -04:00
Emmanuel Vadot
6cafdee71d pkgbase: Track pkg 1.17
With pkg 1.17 the extention is always .pkg, packing_format is also now deprecated.
2021-07-22 10:19:02 +02:00
Alex Richardson
c5d1d88a35 Don't build lldb-tablegen if MK_LLDB=no
Split out from D31060.

Reviewed by:	emaste
2021-07-19 15:05:22 +01:00
John Baldwin
0333fad1b7 Remove svnlite.
Reviewed by:	bcr, imp, emaste
Differential Revision:	https://reviews.freebsd.org/D30737
2021-06-11 14:56:41 -07:00
John Baldwin
e290182bcf Remove 'make update'.
In the CVS days this used be a wrapper around either CVS or CVSup and
used to support updating src, doc, and ports checkouts.  With the move
to subversion this only supported updating src and was itself a
wrapper around 'svn update'.  With Git, users are probably better off
using appropriate Git commands directly to update without needing an
explicit make target as a wrapper.

Reviewed by:	bcr, imp, emaste
Differential Revision:	https://reviews.freebsd.org/D30736
2021-06-11 14:56:28 -07:00
Mitchell Horne
689c7e7975 libpmc: always generate libpmc_events.c
The jevents build tool will create an empty table if it doesn't find any
events, so we can remove the extra $MACHINE_CPUARCH checks.

Reviewed by:	gnn, ray, emaste
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D30531
2021-05-31 17:39:05 -03:00
Marcin Wojtas
c6081dea59 Add elfctl to toolchain.
Add elfctl program to toolchain to allow modifying ELF feature flags
during system build.

Submitted by: Dawid Gorecki <dgr@semihalf.com>
Obtained from: Semihalf
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D29552
2021-05-21 15:32:42 +02:00
Glen Barber
50179c5ec7 Makefile.inc1: unbreak bootstrap when kbdcontrol does not exist
Reviewed by:		arichardson
MFC after:		12 hours
MFC target:		stable/13, releng/13.0
Differential Review:	https://reviews.freebsd.org/D29200
Sponsored by:		Rubicon Communications, LLC ("Netgate")
2021-03-23 20:47:14 -04:00
Baptiste Daroussin
4a847a6ccd Revert "bootstrap: add tic to the bootstrap tools"
Since we don't build and install the terminfo db anymore this is not
needed

This reverts commit b6a51d39e3.
This reverts commit e5e4845959.
This reverts commit 0af562d7e1.
2021-03-18 16:05:20 +01:00
Alex Richardson
e5e4845959 Makefile.inc1 "Avoid duplicate script for target" warning
This happens if -DBOOTSTRAP_ALL_TOOLS or when building on non-FreeBSD.
2021-03-01 13:22:15 +00:00
Baptiste Daroussin
b6a51d39e3 bootstrap: fix a last edit typo
Reported by:	cy
2021-02-25 14:53:00 +01:00
Baptiste Daroussin
0af562d7e1 bootstrap: add tic to the bootstrap tools
tic is necessary early in the build to be able to build the terminfo
database later on.

Tested by:	manu, jbeich
2021-02-25 14:25:28 +01:00
Alex Richardson
5cf6f1c4bc Remove the MK_LIBCPLUSPLUS option
This option has been equivalent to any form of C++ support since libstdc++
was removed. Therefore, replace all MK_LIBCPLUSPLUS uses with MK_CXX.

Reviewed By:	emaste
Differential Revision: https://reviews.freebsd.org/D27974
2021-02-01 09:32:07 +00:00
Kyle Evans
123ae3045d build: remove LIBPTHREAD/LIBTHR build options
WITHOUT_LIBTHR has been broken for a little over five years now, since the
xz 5.2.0 update introduced a hard liblzma dependency on libthr, and building
a useful system without threading support is becoming increasingly more
difficult.

Additionally, in the five plus years that it's been broken more reverse
dependencies have cropped up in libzstd, libsqlite3, and libcrypto (among
others) that make it more and more difficult to reconcile the effort needed
to fix these options.

Remove the broken options.

PR:		252760
Reviewed by:	brooks, emaste, kib
Differential Revision:	https://reviews.freebsd.org/D28263
2021-01-22 12:33:27 -06:00
Glen Barber
a53ce3fc49 Bump CURRENT to 14.0
This one goes to 14.

Approved by:	re (implicit)
Sponsored by:	Rubicon Communications, LLC ("Netgate")
2021-01-21 19:10:07 -05:00
Kyle Evans
26490d9b74 pkgbase: allow update-packages for first-run of packaging
If ${REPODIR}/${PKG_ABI} does not exist when we begin real-update-packages,
skip the comparison with the non-existent previous repository and just
finish the repo off. This allows external scripts to just assume they can
run `update-packages` rather than figuring out if they'd previously run
`packages` for this Version/Arch combo.

PKG_VERSION_FROM_DIR was added so that we could perhaps detect the three
distinct cases:

1.) If the repo has not yet been created, PKG_VERSION_FROM_DIR will be
  empty.
2.) If the repo is in some intermediate state between created and fully
  initialized, PKG_VERSION_FROM_DIR may point to the ABI directory.
3.) If the repo is fully initialized, then PKG_VERSION_FROM_DIR points to
  the latest build to compare to.

Option #2 is explicitly unhandled at the moment, but this is no different
than it was before.

Reviewed-by:	manu
Differential-Revision:	https://reviews.freebsd.org/D28229
2021-01-20 21:58:30 -06:00
Emmanuel Vadot
613fe5321f pkgbase: differentiate package versions for ALPHA/BETA/PRERELEASE/RC phases
The current postfix conversions are:

  CURRENT / STABLE / PRERELEASE, 12.x-CURRENT becomes 12.snapYYYYMMDDhhmmss
  ALPHAx -> .ax, so 11.3-ALPHA1 becomes 11.3.a1.YYYYMMDDhhmmss
  BETAx -> .bx, so 12.1-BETA2 becomes 12.1.b2.YYYYMMDDhhmmss
  RCx -> .rcx, so 13.0-RC3 becomes 13.0.rc3.YYYYMMDDhhmmss
  RELEASE -> (nothing), so 12.1-RELEASE becomes 12.1
  RELEASE-pX -> pX, so 12.1-RELEASE-p1 becomes 12.1p1

Note that for development branches we will start to drop the minor version
component entirely, which more closely matches how these branches are
physically named (stable/NN).

snap is a new prefix that was added to pkg in [0], which is simply a more
verbose version of the current ".s" used.

As noted, build timestamps are also added to ALPHA/BETA/RC versions.  This
is largely irrelevant for re@ snapshots because they will only produce one
set of snapshots for each alpha/beta/rc, but external folks may produce
multiple in that timeframe -- at least for alpha.  For them, it is
imperative that the builds have a differentiating characteristic like this
rather than multiple builds across multiple revisions being versioned
identically.

[0] https://github.com/freebsd/pkg/pull/1929

Reviewed by:	gjb, manu
Submitted by:	rene (original, original version)
Differential Revision:	https://reviews.freebsd.org/D28167
2021-01-18 22:26:22 -06:00
Kyle Evans
95ae95d413 pkgbase: limit PKG_VERSION_FROM calculation to real-update-packages
PKG_ABI is defined in some other targets that do not need to shell out and
calculate PKG_VERSION_FROM. Moreover, it produces extra errors when
bootstrapping an initial pkgbase repo, as the /latest link doesn't exist
yet.
2021-01-18 13:34:54 -06:00
Kyle Evans
da8cc827ae pkgbase: collapse -PRERELEASE into STABLE/CURRENT case for version suffix
-PRERELEASE exists as a given stable/X branch while the next minor version
release is in progress. From a functional standpoint, it should be treated
as -STABLE since it'll typically be a superset of what's included in the
concurrent releng branch.

Reviewed by:	gjb
Differential Revision:	https://reviews.freebsd.org/D28166
2021-01-17 14:08:13 -06:00
Kyle Evans
9af9e7c5cc pkgbase: default to installing all specified kernels
These only get staged off to the side, not installed into the system. Let's
change the default for packages/update-packages to be less surprising.

MFC after:	1 week
Reviewed by:	manu, meena <me igalic co>
Differential Revision:	https://reviews.freebsd.org/D28190
2021-01-17 14:05:33 -06:00
Emmanuel Vadot
03cab14ea3 Revert "pkgbase: differentiate package versions for ALPHA/BETA/PRERELEASE/RC phases"
This reverts commit aef769614f.

While the package names are a bit prettier this confuse pkg about upgrading :
$ pkg version -t 13.0.s2021011313063 13.0.a1
>
$ pkg version -t 13.0.s2021011313063 13.0_ALPHA1
<

Note that the current scheme isn't good when bumping from ALPHA to BETA or
even BETA to RC:
$ pkg version -t 13.0_ALPHA1 13.0_BETA1
=
$ pkg version -t 13.0_BETA1 13.0_RC1
=

But more thoughts have to be put into this renaming.
2021-01-14 14:15:05 +01:00
Emmanuel Vadot
aef769614f pkgbase: differentiate package versions for ALPHA/BETA/PRERELEASE/RC phases
The current postfix conversions are:

    CURRENT / STABLE -> .sYYYYMMDDhhmmss
    ALPHAx -> .ax, so 11.3-ALPHA1 becomes 11.3.a1
    BETAx -> .bx, so 12.1-BETA2 becomes 12.1.b2
    RCx -> .rcx, so 13.0-RC3 becomes 13.0.rc3
    PRERELEASE -> .p, so 11.3-PRERELEASE becomes 11.3.p
    RELEASE -> (nothing), so 12.1-RELEASE becomes 12.1

Submitted by: rene (original version)
Differential Revision: https://reviews.freebsd.org/D20734
2021-01-14 14:00:04 +01:00
Bryan Drewery
44b8b2a00d Makefile.inc1: Avoid using release/Makefile for VERSION.
release/Makefile.inc1 has git executions that were being ran for each of
these lookups.  The results were not needed so just lookup what we want
directly instead.

Reviewed by:	gjb, rlibby, emaste (maybe)
Sponsored by:	Dell EMC
Differential Revision:	https://reviews.freebsd.org/D27643
2021-01-08 14:33:35 -08:00
Alex Richardson
7fa2f2a62f Rename NO_WERROR -> MK_WERROR=no
As suggested in D27598. This also supports MK_WERROR.clang=no and
MK_WERROR.gcc=no to support the existing NO_WERROR.<compiler> uses.

Reviewed By:	brooks
Differential Revision: https://reviews.freebsd.org/D27601
2021-01-07 09:31:03 +00:00
Alex Richardson
94ffff68c8 Stop passing MK_WARNS=no for bootstrap tools
I got annoyed by the number of warnings that the CheriBSD build was
emitting. It turns out that we are emitting lots of warnings during
bootstrap because bootstrap tools are built with the default compiler
flags and ignore the warnings flags that are set in bsd.sys.mk.

Looking at git blame, it appears that MK_WARNS=no has been passed since
rS112869, replacing the -DNO_WERROR option that was added in rS87775.

This commit changes MK_WARNS=no back to -DNO_WERROR. We need to pass
-DNO_WERROR, since the system compiler might have new warnings that we
don't know about yet, and we shouldn't fail the build in that case.

Reviewed By:	imp, brooks
Differential Revision: https://reviews.freebsd.org/D27598
2021-01-07 09:31:03 +00:00
Alex Richardson
0cace0bb0a Add mkimg to the list of bootstrap tools
Having this as part of the bootstrap tools is useful to build disk images
to boot in QEMU (especially when building on a Linux/macOS host where
mkimg is not available). We have been bootstrapping mkimg in CheriBSD for
a long time (using LOCAL_XTOOL_DIRS) but I believe this is also useful
upstream.

Reviewed By:	emaste, brooks
Differential Revision: https://reviews.freebsd.org/D27602
2021-01-07 09:26:21 +00:00
Toomas Soome
f1829643c4 vtfontcvt is part of bootstrap toolset and should always be built
Reported by: Peter Jeremy
2021-01-06 14:52:13 +02:00
Baptiste Daroussin
821aa63a09 ncurses: only keep the version with widechar support
Only keep the widechar version of ncurses as libncursesw.so.9

Keep the old name to avoid breaking the ABI compatibility (the non
widechar version libncurses.so.9 is not binary compatible with
libncursesw.so.9) since all ports and base are already only linking
against the widechar version we can simply remove libncurses.so.9

Since the .9 version only lived in the dev branch and never ended in a
release, it is simply removed and not added to any binary compat
package.

Add symlinks to keep build time compatibility for anyone linking against
-lncurses
2021-01-05 14:01:32 +01:00
Emmanuel Vadot
4401fa9bf1 pkgbase: rename PKGSIGNKEY to PKG_REPO_SIGNING_KEY
this way, it'll be automatically picked up by poudriere
That's quite handy when building pkgbase!

Submitted by:	Mina Galić <me@igalic.co>
Reviewed By: bapt, bcr (manpages)
Differential Revision: https://reviews.freebsd.org/D27690
2021-01-04 16:24:14 +01:00
Ed Maste
fe7dff1759 Remove additional GDB leftovers missed in r368667 2020-12-15 18:12:03 +00:00
Alex Richardson
f7ff7baaf6 crunchgen: fix NULL-deref bug introduced in r364647
While porting over the local changes from CheriBSD for upstreaming, I
accidentally committed a broken version of find_entry_point(): we have to
return NULL if the value is not found instead of a value with
ep->name == NULL, since the checks in main were changed to check ep instead
of ep->name for NULL.

This only matters if the crunched tool cannot be found using normal lookup
and one of the fallback paths is used, so it's unlikely to be triggered
in rescue. However, I noticed that one of our CheriBSD test scripts was
failing to run commands under `su` on minimal disk images where all
binaries are hardlinks to a `cheribsdbox` tool generated with crunchgen.

This also updates the bootstrapping check in Makefile.inc1 to bootstrap
crunchgen up to the next version bump.

Reviewed By:	kevans
Differential Revision: https://reviews.freebsd.org/D27474
2020-12-04 15:53:37 +00:00
Bryan Drewery
80cedb809f Add lists for customizing legacy and bootstrap-tools.
Reviewed by:	arichardson
Sponsored by:	Dell EMC
Differential Revision:	https://reviews.freebsd.org/D27200
2020-11-20 20:11:59 +00:00
Alfredo Dal'Ava Junior
4ae78e70f7 [POWERPC64LE,POWEPCSPE] set default kernel config for powerpc64le and powerpcspe variants
Default KERNCONF for powerpc64le should be GENERIC64, and powerpcspe should
select MPC85XXSPE

Reviewed by:	bdragon,emaste
Sponsored by:	Eldorado Research Institute (eldorado.org.br)
Differential Revision:	https://reviews.freebsd.org/D27257
2020-11-20 18:42:01 +00:00
Alex Richardson
0e55bb7b24 Makefile.inc1: remove no-longer required variable
This variable is unsed since r364760 but I forgot to delete it in that commit.

Reported By:	bdrewery
2020-11-13 13:18:48 +00:00
John-Mark Gurney
8da6fc4d79 fix the docs, this was always wrong... In some cases, DISTDIR is set
automatically by tools via /etc/make.conf, so remind people (me) where
to find where it's set..

It would be nice for someone to document what DISTDIR is better than:
where the file for a distribution gets installed
2020-11-04 23:26:15 +00:00
Emmanuel Vadot
0ac8aa55da pkgbase: Add incremental packages
This adds a new target update-packages which will create the new packages
compared to the last run.

This is how to use it:
At this point we cut a release
$ make buildworld ...
$ make buildkernel
$ make packages

    There is now a PKG_VERSION directory with latest link pointing to it
    Distribute the packages to server

$ something something that update the source tree
$ make buildworld ...
$ make buildkernel
$ make update-packages
You know have a PKG_VERSION directory in the REPODIR and latest link pointing to it.
In PKG_VERSION dir only the packages which differs from the latest run are
named PKG_VERSION, otherwise the old packages are there.

The process is :
Build the new packages in the PKG_VERSION directory
Compare the internal data with the PKG_VERSION_FROM version. The comparison is done
by checking the internal hash of the packages.
By default PKG_VERSION_FROM is set to what the latest link points to.
If the old and new version matches, we rm the new package and cp the old one.

Differential Revision:	https://reviews.freebsd.org/D25984
2020-11-02 18:23:50 +00:00
Ryan Moeller
73577bf01d flua: Add a libjail module
libjail is pretty small, so it makes for a good proof of concept demonstrating
how a system library can be wrapped to create a loadable Lua module for flua.

* Introduce 3lua section for man pages
* Add libjail module

Reviewed by:	kevans, manpages
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D26080
2020-10-24 17:08:59 +00:00
Alex Richardson
3ac62888fc Significantly speed up mkimg_test
It turns out that the majority of the test time for the mkimg tests isn't
mkimg itself but rather the use of jot and hexdump which can be quite slow
on emulated platforms such as QEMU.

On QEMU-RISC-V this reduces the time for `kyua test mkimg_test` from 655
seconds to 200. And for CheriBSD on QEMU-CHERI this saves 4-5 hours (25%
of the time for the entire testsuite!) since jot ends up triggering slow
functions inside the QEMU emulation a lot.

Reviewed By:	lwhsu
Differential Revision: https://reviews.freebsd.org/D26796
2020-10-18 18:35:23 +00:00
Alex Richardson
e5ccad508b Fix build with -DBOOTSTRAP_ALL_TOOLS
sbin/sysctl can no longer be bootstrapped on FreeBSD 12 after r366465,
so create a symlink to the host tool instead of trying to build it.
2020-10-12 10:42:28 +00:00
Kyle Evans
55be47b894 Makefile.inc1: sysent: allow subordinate sysent targets to run in parallel
makesyscalls.lua (and indeed makesyscalls.sh) are both safe to be run in
parallel, so let's do it.

This is a trivial difference because runtime per-target is pretty small, but
I like seeing it run in parallel when my muscle memory types `make -sj4`.

Reviewed by:	brooks, emaste
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D26594
2020-09-30 00:47:57 +00:00
Brandon Bergren
b75abea4d0 [PowerPC64LE] Set up powerpc.powerpc64le architecture
This is the initial set up for PowerPC64LE.

The current plan is for this arch to remain experimental for FreeBSD 13.

This started as a weekend learning project for me and kinda snowballed from
there.

(More to follow momentarily.)

Reviewed by:	imp (earlier version), emaste
Sponsored by:	Tag1 Consulting, Inc.
Differential Revision:	https://reviews.freebsd.org/D26399
2020-09-22 23:49:30 +00:00
Alex Richardson
6129f33eb2 Prefer bootstrapped tools when running certctl.sh
Otherwise we get lots of warnings when building on Linux/macOS during
installworld:
Scanning /local/scratch/alr48/cheri/output/freebsd-x86/usr/share/certs/blacklisted for certificates...
install: invalid option -- 'U'
Try 'install --help' for more information.
install: invalid option -- 'U'
....

Reviewed By:	kevans
Differential Revision: https://reviews.freebsd.org/D26481
2020-09-21 09:03:32 +00:00
Toomas Soome
e307eb94ae loader: zfs should support bootonce an nextboot
bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.

By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.

bootonce dataset name is recorded in boot pool labels, bootenv area.

in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.

bootonce and nextboot features are usable in both UEFI and BIOS boot.

To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).

At this time, only lua loader is updated.

Sponsored by:	Netflix, Klara Inc.
Differential Revision:	https://reviews.freebsd.org/D25512
2020-09-21 09:01:10 +00:00
Kyle Evans
fe815331bb build: provide a default WARNS for all in-tree builds
The current default is provided in various Makefile.inc in some top-level
directories and covers a good portion of the tree, but doesn't cover parts
of the build a little deeper (e.g. libcasper).

Provide a default in src.sys.mk and set WARNS to it in bsd.sys.mk if that
variable is defined. This lets us relatively cleanly provide a default WARNS
no matter where you're building in the src tree without breaking things
outside of the tree.

Crunchgen has been updated as a bootstrap tool to work on this change
because it needs r365605 at a minimum to succeed. The cleanup necessary to
successfully walk over this change on WITHOUT_CLEAN builds has been added.

There is a supplemental project to this to list all of the warnings that are
encountered when the environment has WARNS=6 NO_WERROR=yes:
https://warns.kevans.dev -- this project will hopefully eventually go away
in favor of CI doing a much better job than it.

Reviewed by:	emaste, brooks, ngie (all earlier version)
Reviewed by:	emaste, arichardson (depend-cleanup.sh change)
Differential Revision:	https://reviews.freebsd.org/D26455
2020-09-18 17:17:46 +00:00
Kyle Evans
ca4b73c348 Promote the installworld certctl rehash to distributeworld
Contrary to my belief, installworld is not sufficient for getting certs
installed into VM images. Promote the rehash to both installworld and
distributeworld (notably: not stageworld) and rehash the base distdir so we
end up with /etc/ssl/certs populated in the base dist archive. A future
commit will remove the rehash from bsdinstall, which doesn't really need to
happen if they're installed into base.txz.

While here, fix a minor typo: s/CERTCLTFLAGS/CERTCTLFLAGS/

MFC after:	1 week
2020-09-17 15:58:42 +00:00
Kyle Evans
185e8af021 installworld: run certctl rehash after installation completes
This was originally introduced back in r360833, and subsequently reverted
because it was broken for -DNO_ROOT builds and it may not have been the
correct place for it.

While debatably this may still not be 'the correct place,' it's much cleaner
than scattering rehashes all throughout the tree. brooks has fixed the issue
with -DNO_ROOT by properly writing to the METALOG in r361397.

Do note that this is different than what was originally committed; brooks
had revisions in D24932 that made it actually use the revised unprivileged
mode and write to METALOG, along with being a little more friendly to
foreign crossbuilds and just using the certctl in-tree.

With this change, I believe we should now have a populated /etc/ssl/certs in
the VM images.

MFC after:	1 week
2020-09-17 02:18:21 +00:00
Ed Maste
073e409487 Makefile.inc1: remove more old stale depend hacks
Current stale dependency hacks are in tools/build/depend-cleanup.sh.
These ones were almost a year old; remove them from Makefile.inc1.
2020-09-13 23:05:19 +00:00
Alex Richardson
b7b5bdba96 Ensure that the makewhatis symlink is added in the bootstrap-tools stage
We currently set MK_MAN=no in $BSARGS so MK_MAN_UTILS will also be false
which means that the makewhatis symlink will not be created.
This change fixes the build when using both -DBUILD_WITH_STRICT_TMPPATH and
-DBOOTSTRAP_ALL_TOOLS.

Tested by:	andrew
Differential Revision: https://reviews.freebsd.org/D16761
2020-09-10 15:37:29 +00:00
Ed Maste
7576679986 Add WITH_/WITHOUT_CLEAN option to replace NO_CLEAN
This allows use of the standard src.conf configuration for controlling
whether the tree is cleaned before build or not.  The default is still
to clean.

Setting either NOCLEAN or NO_CLEAN will mention the new src.conf option.
NOCLEAN remains a .warning, while for now NO_CLEAN is .info.

Reviewed by:	bdrewery (earlier version)
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D22762
2020-09-08 00:44:35 +00:00
Ed Maste
30d46e454a Makefile.inc1: comment .endif to ease finding matching .if 2020-09-01 00:14:40 +00:00
Glen Barber
e25152834c MFH
Sponsored by:	Rubicon Communications, LLC (netgate.com)
2020-08-27 20:25:33 +00:00
Ryan Moeller
cd568e2b1b libzfs: Also add the crypto dependency to Makefile.inc1
Reported by:	kevans
Discussed with:	kevans
Sponsored by:	iXsystems, Inc.
2020-08-27 13:26:36 +00:00
Alex Richardson
3ce13dbc23 Use bootstrapped install(1) install of tools/install.sh in world stage
This should be noticeably faster due to fewer processes being forked and
also handles other flags such as -S or writing to METALOG.

Reviewed By:	brooks
Differential Revision: https://reviews.freebsd.org/D26039
2020-08-25 13:30:14 +00:00
Alex Richardson
eb51ce8efb Fix running the builddtb target on a noexec file system
Obtained from:	CheriBSD
2020-08-25 13:30:09 +00:00
Alex Richardson
5bb9250e0a Add necessary Makefile.inc1 infrastructure for building on non-FreeBSD
The most awkward bit in this patch is the bootstrapping of m4:
We can't simply use the host version of m4 since that is not compatible
with the flags passed by lex (at least on macOS, possibly also on Linux).
Therefore we need to bootstrap m4, but lex needs m4 to build and m4 also
depends on lex (which needs m4 to generate any files). To work around this
cyclic dependency we can build a bootstrap version of m4 (with pre-generated
files) then use that to build the real m4.

This patch also changes the xz/unxz/dd tools to always use the host version
since the version in the source tree cannot easily be bootstrapped on macOS
or Linux.

Reviewed By:	brooks, imp (earlier version)
Differential Revision: https://reviews.freebsd.org/D25992
2020-08-25 13:29:57 +00:00
Matt Macy
9e5787d228 Merge OpenZFS support in to HEAD.
The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
  project quotas, encrypted datasets,
  allocation classes, vectorized raidz,
  vectorized checksums, various command line
  improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D25872
2020-08-25 02:21:27 +00:00
Warner Losh
0e533c72bc When copying over the binaries, use '-p' to preserve date/time
Although I can't reproduce it, others are seeing different lex/yacc
programs always regenerated after my change to copy rather than
symlink the files. The reported fix is to add '-p' to the copies.
Since it doesn't hurt, go head and add it, though the reasons for
this mattering remain at best obscure and poorly articulated.
2020-08-24 16:06:11 +00:00
Alex Richardson
ebae797c82 Also print number of available CPUs on Linux
Without this change the buildworld/buildkernel epilogue looks like this:
>>> World built in 249 seconds, sysctl: cannot stat /proc/sys/hw/ncpu: No such file or directory
ncpu: , make -j72.

Reviewed By:	emaste, bdrewery
Differential Revision: https://reviews.freebsd.org/D26056
2020-08-24 09:20:38 +00:00
Alex Richardson
0f31fdf253 Pass the installworld install(1) flags to make buildenv
This ensure that running make install inside buildenv correctly includes
the METALOG flags when building with -DNO_ROOT.

Reviewed By:	brooks
Differential Revision: https://reviews.freebsd.org/D26038
2020-08-24 09:20:13 +00:00
Bjoern A. Zeeb
530134d291 After the clang/llvm version 11 import LLD_VERSION is no longer used
upstream so Version.inc now only defines LLD_VERSION_STRING.

This breaks the WANT_LINKER_VERSION magic and might lead to us building
more than needed (e.g., for croos-tools).

Change the awk script to parse LLD_VERSION_STRING instead of LLD_VERSION,
which not only unbreaks the current situation but should also be backwards
compatible as dim points out.

PR:		248818
Reviewed by:	emaste, dim (seems right and the way to go)
MFC after:	4 weeks
X-MFC before:	364284
2020-08-21 22:31:45 +00:00
Glen Barber
440cec3faf MFH
Sponsored by:	Rubicon Communications, LLC (netgate.com)
2020-08-12 17:16:26 +00:00
Alex Richardson
f57b27e3f8 Fix -DBUILD_WITH_STRICT_TMPPATH dtrace builds
Some of the scripts used for libdtrace invoke nawk instead of awk
(for example cddl/contrib/opensolaris/lib/libdtrace/common/mknames.sh).
When bootstrapping all tools, we get the nawk -> awk link while building
usr.bin/awk, but when linking/copying the dependencies from the host we
were only adding awk but not nawk.

This was silently generating invalid files when building libdtrace with
BUILD_WITH_STRICT_TMPPATH=1 since those scripts invoke nawk instead of
awk. In addition to adding the missing link this commit also adds
set -e to those scripts to catch errors like this in the future.

Reviewed By:	markj, emaste
Differential Revision: https://reviews.freebsd.org/D26025
2020-08-11 16:46:48 +00:00
Alex Richardson
1a18ab420b Allow overriding the tool used for stripping binaries
Since the make variable STRIP is already used for other purposes, this
uses STRIPBIN (which is also used for the same purpose by install(1).
This allows using LLVM objcopy to strip binaries instead of the in-tree
elftoolchain objcopy. We make use of this in CheriBSD since passing
binaries generated by our toolchain to elftoolchain strip sometimes results
in assertion failures.

This allows working around https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248516
by specifying STRIPBIN=/path/to/llvm-strip

Obtained from:	CheriBSD
Reviewed By:	emaste, brooks
Differential Revision: https://reviews.freebsd.org/D25988
2020-08-11 16:46:27 +00:00
Emmanuel Vadot
a90022d4d1 pkgbase: Replace / with | for PKG_WWW
PKG_WWW contain / char, replace the delimiter by a '|'.

Reported by:	0mp
2020-08-11 10:07:59 +00:00
Emmanuel Vadot
82087d4b68 pkgbase: Add PKG_NAME_PREFIX, PKG_MAINTAINER and PKG_WWW
This is useful for downstream users to customize the packages.
Reviewed by:	emaste
Differential Revision:	https://reviews.freebsd.org/D26019
2020-08-11 08:42:24 +00:00
Warner Losh
33176cdc87 The practice of creating symbolic links is somewhat fragile. Always
make copies instead.

There's too many times that we can't run the new binaries with old
libraries. Making the links when things are known to be 'safe' is a
nice optimization, but a copy of all the binaries is only 30MB, so
saving the copies at the cost of increased support when new symbols
are added and used as part of the bootstrap seems to be unwise.

There may be additional optimizations possible here, especially for
!FreeBSD hosts. However, that's beyond the scope of the problem I'm
trying to fix with make failing mid-way through an installworld across
change r363679. This optimization there caused us to run a new binary
with an old library once a new make was installed due to the symbolic
link. One could just copy make, but then other binaries fail as well,
so rather than play whack-a-mole, I opted to take us back to the old
way.  Before r340157 or so we did copies (thogh of a lot fewer
artifacts), and we didn't have issues like this.

Reviewed by: arichards@
Differential Revision: https://reviews.freebsd.org/D25967
2020-08-07 16:26:56 +00:00
Alex Richardson
0d834e0f4d Fix duplicate assignment of _localedef in Makefile.inc1
The same .if exists a few lines below.
2020-08-07 16:04:15 +00:00
Emmanuel Vadot
70fc0cc387 pkgbase: Add the possibility to choose the output dir
The output dir is set to ${REPODIR}/${PKG_ABI}/${PKG_VERSION} now.
Add the possibility to specify the last componant and set it by default
to ${PKG_VERSION} as before.
This is useful for tests and also for building packages with the same
PKG_VERSION provided to check differences.
2020-08-06 16:14:43 +00:00
Emmanuel Vadot
d0327929ab pkgbase: Add a new variable PKG_TIMESTAMP
libarchive uses the SOURCE_DATE_EPOCH env variable to set the date of file
in an archive, this is useful for reproducibility.
Add a variable name PKG_TIMESTAMP that take a epoch time and set SOURCE_DATE_EPOCH
to this. By default it is the current time so no changes here.
2020-08-06 16:13:54 +00:00
Kyle Evans
eb578fec7f Ensure libregex is built in time for googletest
In lib/Makefile, we document the dependency with SUBDIR_DEPEND

For buildworld orchestration, just prebuild libregex if GOOGLETEST is
enabled. googletest will get built in a later pass.
2020-08-04 03:43:28 +00:00
Glen Barber
40056899f6 Move GIT_CMD a few lines down, so it is not in between SVN_*
lines.

Sponsored by:	Rubicon Communications, LLC (netgate.com)
2020-07-16 20:44:04 +00:00
Glen Barber
ff1031c1b7 Add GIT_CMD, similar to SVN_CMD.
Sponsored by:	Rubicon Communications, LLC (netgate.com)
2020-07-16 20:36:22 +00:00
Conrad Meyer
0a9ab9f549 Add WITH_CLANG_FORMAT option
clang-format is enabled conditional on either WITH_CLANG_EXTRAS or
WITH_CLANG_FORMAT.  Some sources in libclang are build conditional on
either rule, and obviously the clang-format binary itself depends on the
rule.

clang-format could still use a manual page.

Reviewed by:	emaste
Differential Revision:	https://reviews.freebsd.org/D25427
2020-06-24 17:03:42 +00:00
Kyle Evans
a19aa4f704 Squash liblzma build race
As of r362452, liblzma depends on libmd but the buildworld build order
hadn't been amended to document the new dependency.

Reported by:	jenkins via freqlabs
X-MFC-With:	r362452
2020-06-22 03:44:01 +00:00
Ryan Moeller
94a8266684 flua: add ucl library
libucl comes with a Lua library binding.  Build it into flua.

This lets us parse/generate config files in the various formats supported by
libucl with flua.  For example, the following script will detect the format of
an object written to stdin as one of UCL config, JSON, or YAML and write it to
stdout as pretty-printed JSON:

local ucl = require('ucl')
local parser = ucl.parser()
parser:parse_string(io.read('*a'))
local obj = parser:get_object()
print(ucl.to_format(obj, 'json'))

Reviewed by:	kevans, pstef
Approved by:	mmacy (mentor)
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D25009
2020-06-22 03:14:43 +00:00
Ed Maste
74e8d41e0a Retire BINUTILS and BINUTILS_BOOTSTRAP options
As of r361857 all BINUTILS options are disabled by default - ports
have been changed to depend on binutils if they require GNU as, and
all base system assembly files have been switched to use Clang's
integrated assembler.

Relnotes:	Yes
Sponsored by:	The FreeBSD Foundation
2020-06-07 00:07:21 +00:00
Ed Maste
5843b6e7b2 Makefile.inc1: remove stale dependency cleanup from 2018 2020-06-05 20:18:55 +00:00
Ed Maste
694034e227 Makefile.inc1: remove BINUTILS_BOOTSTRAP linker support
As of r359347 BINUTILS_BOOTSTRAP does not include the bfd linker.
2020-05-30 16:20:18 +00:00
Ed Maste
697b271da9 pkgbase: use -dev,-dbg instead of -development,-debug
-development is long and awkward, and is also inconsistent with prior art
from the Linux world, which uses -dev (Debian) or -devel (Red Hat).  Follow
the Debian convention, and similarly for debug info packages.

Also remove redundant pkgbase development tag from includes.  We already tag
include files with package=runtime,dev; there is no need to separately tag
them as dev.

Discussed with:	bapt
Reviewed by:	manu
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D24139
2020-05-20 19:45:22 +00:00
Kyle Evans
20076f97fd Revert r360833, r360882: certctl rehash in installworld
This was solving the correct-ish problem in the wrong place. Noted by
brooks; while he didn't request a prompt revert, doing so now will
facilitate proper testing for the revised version of this.
2020-05-18 01:48:55 +00:00
Kyle Evans
a93817e04f buildworld: add back in missing semicolon
if foo; then blah else blah; fi has a drastically different meaning than
if foo; then blah; else blah; fi. Fix it.

Reported by:	0mp
X-MFC-With:	r360833
2020-05-10 20:28:38 +00:00
Kyle Evans
70868d48e8 installworld: attempt a certctl rehash at the tail end
This can be run as root or normal user with no problem; if they hadn't
twisted the WITHOUT_CAROOT knob, we'll attempt to use the host certctl to
rehash the DESTDIR. This would allow one to build systems WITHOUT_OPENSSL +
WITH_CAROOT with a populated /etc/ssl that they can then use with an
appropriate *ssl from somewhere else.

Cross-builds are fine because this will always use the host certctl, or just
nag if it's missing and it wasn't a WITHOUT_CAROOT build.

MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D24641
2020-05-09 02:01:29 +00:00
Dimitry Andric
61fbd61b59 Fix race between prebuilding libsbuf and libgeom
The latter needs the former, but with a multi-job build on a fast
machine, the race is sometimes lost. This leads to "ld: error: unable to
find library -lsbuf", when linking libgeom.so.

Submitted by:	kevans
MFC after:	3 days
2020-04-25 20:24:41 +00:00
Kyle Evans
6fe257c780 Move shm_open dependency cleanup into a new home
r359461 introduced this nifty script to centralize these things, so add
shm_open.c there to remove a total of one (1) bad example from
Makefile.inc1.

Looked over by:	emaste
2020-04-13 19:50:47 +00:00
Maxim Sobolev
a8b8edb25e Normalize deployment tools usage and definitions by putting into one place
instead of sprinkling them out over many disjoint files. This is a follow-up
to achieve the same goal in an incomplete rev.348521.

Approved by:	imp
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D20520
2020-04-07 02:46:22 +00:00
Brooks Davis
f382bac49b Fix compilation with upstream clang builtin headers.
By using -nobuiltininc and adding the clang builtin headers resource dir
to the end of the compiler header search path, we can still find headers
such as immintrin.h but find the FreeBSD version of stddef.h/stdarg.h/..
first.

This is a workaround until we are able to settle on and complete a plan
to harmonize guard macros with LLVM.  We've mostly worked out this on
FreeBSD systems by removing select headers from the installed set of
devel/llvm*, but that isn't a good solution for cross build.

Submitted by:	arichardson
Obtained from:	CheriBSD
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D17002
2020-04-06 23:38:46 +00:00
Brooks Davis
e40fa28f1a Pass the real value of MK_TESTS_SUPPORT to distribution.
This allows kyua's config file to actually be installed.

Reported by:	arichardson
Reviewed by:	arichardson
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D24268
2020-04-03 16:19:07 +00:00
Ed Maste
4dad87a498 add shell script for stale dependency hack
It's rather awkward to debug issues with the dependency cleanup hacks
when implemented via make.  Add a cleanup shell script and move the
libomp hack there as an initial example.

Reviewed by:	brooks
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D24228
2020-03-30 20:20:15 +00:00
Brooks Davis
c42a67671b Relax existence check of ${LOCALBASE}/bin/kyua
After the base kyua import the testsuite can utilize the in-base kyua
just fine.

Submitted by:	Dries Michiels
Differential Revision:	https://reviews.freebsd.org/D24230
2020-03-30 20:04:55 +00:00
Brooks Davis
be2af8fbf4 Use the real value of MK_TESTS_SUPPORT in _libraries.
We need to build kyua libraries for kyua.  Because we set MK_TESTS=no,
we can't not set MK_TESTS_SUPPORT=${MK_TESTS_SUPPORT} because the latter
defaults MK_TESTS_SUPPORT to no.

This fixes WITHOUT_TESTS + WITH_TESTS_SUPPORT builds.

Reported by:	Ruslan Garipov
Reviewed by:	emaste
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D24206
2020-03-27 23:27:54 +00:00
Emmanuel Vadot
eb61f7bdf2 Stop building libl and liby
Those libs aren't needed anymore so stop building them for the bootstrap tools
or the compat libs.

Reviewed by:	bapt, emaste
X-Differential Revision:	D24171
2020-03-26 08:23:09 +00:00
Mitchell Horne
52de22cae7 Makefile.inc1: override MACHINE for native-xtools
For the final step of the native-xtools target, "everything" is built
with TARGET and TARGET_ARCH set to the architecture we wish to
cross-build for. However, CROSSENV overwrites the values of MACHINE and
MACHINE_ARCH, setting them to be identical to TARGET and TARGET_ARCH.
For native-xtools this is undesirable since we are building binaries to
run on the host architecture, that can cross compile for the target
architecture.

When building native-xtools for RISC-V, this issue manifests as an invalid
argument for "-march". The compiler is invoked with the target triple of the
host architecture, but the CFLAGS inherited from bsd.cpu.mk are that of the
target architecture.

Reviewed by:	imp, bdrewery
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D23838
2020-03-24 23:25:54 +00:00
Ed Maste
f1af40b54b Makefile.inc1: remove old (2018) stale dependency hacks
We handle a number of stale dependency issues on an ad-hoc basis, in
order to support ongoing NO_CLEAN builds.  These hacks do not need to
be maintained indefinitely; now remove those for issues that are more
than a year old.

Sponsored by:	The FreeBSD Foundation
2020-03-24 21:37:34 +00:00
Ed Maste
c260dbfb47 Makefile.inc1: set DB_FROM_SRC also for stageworld
stageworld, invoked from e.g. `make packages`, shouldn't rely on the
build host's users/groups.

Sponsored by:	The FreeBSD Foundation
2020-03-22 03:10:31 +00:00