Commit Graph

1662 Commits

Author SHA1 Message Date
Alex Richardson
f57b27e3f8 Fix -DBUILD_WITH_STRICT_TMPPATH dtrace builds
Some of the scripts used for libdtrace invoke nawk instead of awk
(for example cddl/contrib/opensolaris/lib/libdtrace/common/mknames.sh).
When bootstrapping all tools, we get the nawk -> awk link while building
usr.bin/awk, but when linking/copying the dependencies from the host we
were only adding awk but not nawk.

This was silently generating invalid files when building libdtrace with
BUILD_WITH_STRICT_TMPPATH=1 since those scripts invoke nawk instead of
awk. In addition to adding the missing link this commit also adds
set -e to those scripts to catch errors like this in the future.

Reviewed By:	markj, emaste
Differential Revision: https://reviews.freebsd.org/D26025
2020-08-11 16:46:48 +00:00
Alex Richardson
1a18ab420b Allow overriding the tool used for stripping binaries
Since the make variable STRIP is already used for other purposes, this
uses STRIPBIN (which is also used for the same purpose by install(1).
This allows using LLVM objcopy to strip binaries instead of the in-tree
elftoolchain objcopy. We make use of this in CheriBSD since passing
binaries generated by our toolchain to elftoolchain strip sometimes results
in assertion failures.

This allows working around https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248516
by specifying STRIPBIN=/path/to/llvm-strip

Obtained from:	CheriBSD
Reviewed By:	emaste, brooks
Differential Revision: https://reviews.freebsd.org/D25988
2020-08-11 16:46:27 +00:00
Emmanuel Vadot
a90022d4d1 pkgbase: Replace / with | for PKG_WWW
PKG_WWW contain / char, replace the delimiter by a '|'.

Reported by:	0mp
2020-08-11 10:07:59 +00:00
Emmanuel Vadot
82087d4b68 pkgbase: Add PKG_NAME_PREFIX, PKG_MAINTAINER and PKG_WWW
This is useful for downstream users to customize the packages.
Reviewed by:	emaste
Differential Revision:	https://reviews.freebsd.org/D26019
2020-08-11 08:42:24 +00:00
Warner Losh
33176cdc87 The practice of creating symbolic links is somewhat fragile. Always
make copies instead.

There's too many times that we can't run the new binaries with old
libraries. Making the links when things are known to be 'safe' is a
nice optimization, but a copy of all the binaries is only 30MB, so
saving the copies at the cost of increased support when new symbols
are added and used as part of the bootstrap seems to be unwise.

There may be additional optimizations possible here, especially for
!FreeBSD hosts. However, that's beyond the scope of the problem I'm
trying to fix with make failing mid-way through an installworld across
change r363679. This optimization there caused us to run a new binary
with an old library once a new make was installed due to the symbolic
link. One could just copy make, but then other binaries fail as well,
so rather than play whack-a-mole, I opted to take us back to the old
way.  Before r340157 or so we did copies (thogh of a lot fewer
artifacts), and we didn't have issues like this.

Reviewed by: arichards@
Differential Revision: https://reviews.freebsd.org/D25967
2020-08-07 16:26:56 +00:00
Alex Richardson
0d834e0f4d Fix duplicate assignment of _localedef in Makefile.inc1
The same .if exists a few lines below.
2020-08-07 16:04:15 +00:00
Emmanuel Vadot
70fc0cc387 pkgbase: Add the possibility to choose the output dir
The output dir is set to ${REPODIR}/${PKG_ABI}/${PKG_VERSION} now.
Add the possibility to specify the last componant and set it by default
to ${PKG_VERSION} as before.
This is useful for tests and also for building packages with the same
PKG_VERSION provided to check differences.
2020-08-06 16:14:43 +00:00
Emmanuel Vadot
d0327929ab pkgbase: Add a new variable PKG_TIMESTAMP
libarchive uses the SOURCE_DATE_EPOCH env variable to set the date of file
in an archive, this is useful for reproducibility.
Add a variable name PKG_TIMESTAMP that take a epoch time and set SOURCE_DATE_EPOCH
to this. By default it is the current time so no changes here.
2020-08-06 16:13:54 +00:00
Kyle Evans
eb578fec7f Ensure libregex is built in time for googletest
In lib/Makefile, we document the dependency with SUBDIR_DEPEND

For buildworld orchestration, just prebuild libregex if GOOGLETEST is
enabled. googletest will get built in a later pass.
2020-08-04 03:43:28 +00:00
Conrad Meyer
0a9ab9f549 Add WITH_CLANG_FORMAT option
clang-format is enabled conditional on either WITH_CLANG_EXTRAS or
WITH_CLANG_FORMAT.  Some sources in libclang are build conditional on
either rule, and obviously the clang-format binary itself depends on the
rule.

clang-format could still use a manual page.

Reviewed by:	emaste
Differential Revision:	https://reviews.freebsd.org/D25427
2020-06-24 17:03:42 +00:00
Kyle Evans
a19aa4f704 Squash liblzma build race
As of r362452, liblzma depends on libmd but the buildworld build order
hadn't been amended to document the new dependency.

Reported by:	jenkins via freqlabs
X-MFC-With:	r362452
2020-06-22 03:44:01 +00:00
Ryan Moeller
94a8266684 flua: add ucl library
libucl comes with a Lua library binding.  Build it into flua.

This lets us parse/generate config files in the various formats supported by
libucl with flua.  For example, the following script will detect the format of
an object written to stdin as one of UCL config, JSON, or YAML and write it to
stdout as pretty-printed JSON:

local ucl = require('ucl')
local parser = ucl.parser()
parser:parse_string(io.read('*a'))
local obj = parser:get_object()
print(ucl.to_format(obj, 'json'))

Reviewed by:	kevans, pstef
Approved by:	mmacy (mentor)
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D25009
2020-06-22 03:14:43 +00:00
Ed Maste
74e8d41e0a Retire BINUTILS and BINUTILS_BOOTSTRAP options
As of r361857 all BINUTILS options are disabled by default - ports
have been changed to depend on binutils if they require GNU as, and
all base system assembly files have been switched to use Clang's
integrated assembler.

Relnotes:	Yes
Sponsored by:	The FreeBSD Foundation
2020-06-07 00:07:21 +00:00
Ed Maste
5843b6e7b2 Makefile.inc1: remove stale dependency cleanup from 2018 2020-06-05 20:18:55 +00:00
Ed Maste
694034e227 Makefile.inc1: remove BINUTILS_BOOTSTRAP linker support
As of r359347 BINUTILS_BOOTSTRAP does not include the bfd linker.
2020-05-30 16:20:18 +00:00
Ed Maste
697b271da9 pkgbase: use -dev,-dbg instead of -development,-debug
-development is long and awkward, and is also inconsistent with prior art
from the Linux world, which uses -dev (Debian) or -devel (Red Hat).  Follow
the Debian convention, and similarly for debug info packages.

Also remove redundant pkgbase development tag from includes.  We already tag
include files with package=runtime,dev; there is no need to separately tag
them as dev.

Discussed with:	bapt
Reviewed by:	manu
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D24139
2020-05-20 19:45:22 +00:00
Kyle Evans
20076f97fd Revert r360833, r360882: certctl rehash in installworld
This was solving the correct-ish problem in the wrong place. Noted by
brooks; while he didn't request a prompt revert, doing so now will
facilitate proper testing for the revised version of this.
2020-05-18 01:48:55 +00:00
Kyle Evans
a93817e04f buildworld: add back in missing semicolon
if foo; then blah else blah; fi has a drastically different meaning than
if foo; then blah; else blah; fi. Fix it.

Reported by:	0mp
X-MFC-With:	r360833
2020-05-10 20:28:38 +00:00
Kyle Evans
70868d48e8 installworld: attempt a certctl rehash at the tail end
This can be run as root or normal user with no problem; if they hadn't
twisted the WITHOUT_CAROOT knob, we'll attempt to use the host certctl to
rehash the DESTDIR. This would allow one to build systems WITHOUT_OPENSSL +
WITH_CAROOT with a populated /etc/ssl that they can then use with an
appropriate *ssl from somewhere else.

Cross-builds are fine because this will always use the host certctl, or just
nag if it's missing and it wasn't a WITHOUT_CAROOT build.

MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D24641
2020-05-09 02:01:29 +00:00
Dimitry Andric
61fbd61b59 Fix race between prebuilding libsbuf and libgeom
The latter needs the former, but with a multi-job build on a fast
machine, the race is sometimes lost. This leads to "ld: error: unable to
find library -lsbuf", when linking libgeom.so.

Submitted by:	kevans
MFC after:	3 days
2020-04-25 20:24:41 +00:00
Kyle Evans
6fe257c780 Move shm_open dependency cleanup into a new home
r359461 introduced this nifty script to centralize these things, so add
shm_open.c there to remove a total of one (1) bad example from
Makefile.inc1.

Looked over by:	emaste
2020-04-13 19:50:47 +00:00
Maxim Sobolev
a8b8edb25e Normalize deployment tools usage and definitions by putting into one place
instead of sprinkling them out over many disjoint files. This is a follow-up
to achieve the same goal in an incomplete rev.348521.

Approved by:	imp
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D20520
2020-04-07 02:46:22 +00:00
Brooks Davis
f382bac49b Fix compilation with upstream clang builtin headers.
By using -nobuiltininc and adding the clang builtin headers resource dir
to the end of the compiler header search path, we can still find headers
such as immintrin.h but find the FreeBSD version of stddef.h/stdarg.h/..
first.

This is a workaround until we are able to settle on and complete a plan
to harmonize guard macros with LLVM.  We've mostly worked out this on
FreeBSD systems by removing select headers from the installed set of
devel/llvm*, but that isn't a good solution for cross build.

Submitted by:	arichardson
Obtained from:	CheriBSD
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D17002
2020-04-06 23:38:46 +00:00
Brooks Davis
e40fa28f1a Pass the real value of MK_TESTS_SUPPORT to distribution.
This allows kyua's config file to actually be installed.

Reported by:	arichardson
Reviewed by:	arichardson
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D24268
2020-04-03 16:19:07 +00:00
Ed Maste
4dad87a498 add shell script for stale dependency hack
It's rather awkward to debug issues with the dependency cleanup hacks
when implemented via make.  Add a cleanup shell script and move the
libomp hack there as an initial example.

Reviewed by:	brooks
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D24228
2020-03-30 20:20:15 +00:00
Brooks Davis
c42a67671b Relax existence check of ${LOCALBASE}/bin/kyua
After the base kyua import the testsuite can utilize the in-base kyua
just fine.

Submitted by:	Dries Michiels
Differential Revision:	https://reviews.freebsd.org/D24230
2020-03-30 20:04:55 +00:00
Brooks Davis
be2af8fbf4 Use the real value of MK_TESTS_SUPPORT in _libraries.
We need to build kyua libraries for kyua.  Because we set MK_TESTS=no,
we can't not set MK_TESTS_SUPPORT=${MK_TESTS_SUPPORT} because the latter
defaults MK_TESTS_SUPPORT to no.

This fixes WITHOUT_TESTS + WITH_TESTS_SUPPORT builds.

Reported by:	Ruslan Garipov
Reviewed by:	emaste
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D24206
2020-03-27 23:27:54 +00:00
Emmanuel Vadot
eb61f7bdf2 Stop building libl and liby
Those libs aren't needed anymore so stop building them for the bootstrap tools
or the compat libs.

Reviewed by:	bapt, emaste
X-Differential Revision:	D24171
2020-03-26 08:23:09 +00:00
Mitchell Horne
52de22cae7 Makefile.inc1: override MACHINE for native-xtools
For the final step of the native-xtools target, "everything" is built
with TARGET and TARGET_ARCH set to the architecture we wish to
cross-build for. However, CROSSENV overwrites the values of MACHINE and
MACHINE_ARCH, setting them to be identical to TARGET and TARGET_ARCH.
For native-xtools this is undesirable since we are building binaries to
run on the host architecture, that can cross compile for the target
architecture.

When building native-xtools for RISC-V, this issue manifests as an invalid
argument for "-march". The compiler is invoked with the target triple of the
host architecture, but the CFLAGS inherited from bsd.cpu.mk are that of the
target architecture.

Reviewed by:	imp, bdrewery
MFC after:	2 weeks
Differential Revision:	https://reviews.freebsd.org/D23838
2020-03-24 23:25:54 +00:00
Ed Maste
f1af40b54b Makefile.inc1: remove old (2018) stale dependency hacks
We handle a number of stale dependency issues on an ad-hoc basis, in
order to support ongoing NO_CLEAN builds.  These hacks do not need to
be maintained indefinitely; now remove those for issues that are more
than a year old.

Sponsored by:	The FreeBSD Foundation
2020-03-24 21:37:34 +00:00
Ed Maste
c260dbfb47 Makefile.inc1: set DB_FROM_SRC also for stageworld
stageworld, invoked from e.g. `make packages`, shouldn't rely on the
build host's users/groups.

Sponsored by:	The FreeBSD Foundation
2020-03-22 03:10:31 +00:00
Ed Maste
dadc342d07 Revert r359083, fixed properly by r359088
r359083 introduced a workaround for stale libomp dependencies during a
regular (no -DNO_CLEAN) buildworld.  r359088 addressed the reason the
clean step missed libomp, so revert the workaround.

Sponsored by:	The FreeBSD Foundation
2020-03-18 21:20:10 +00:00
Ed Maste
0d02682c96 invoke _cleanobj_fast_depend_hack unconditionally
Apparently make ${CLEANDIR} is leaving stale entries in .depend files;
for now invoke the hacky cleanup in both the -DNO_CLEAN and normal
(no -DNO_CLEAN) cases.

In collaboration with:	dim
Sponsored by:	The FreeBSD Foundation
2020-03-18 20:12:46 +00:00
Brooks Davis
564b9ff2a7 Add an internal liblua and use it in flua.
The new liblua will be used in a forthcoming import of kyua.

Reviewed by:	kevans
Obtained from:	CheriBSD
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D24090
2020-03-17 17:28:12 +00:00
Ed Maste
dfcef1de85 Makefile.inc1: add a note when deleting stale dependencies
We have ad-hoc stale dependency handling in Makefile.inc1 to handle the
cases where file extensions change, but it appears that some cases are
not functional.  Add a note when about to clean stale deps to help
when investigating failure reports.

Sponsored by:	The FreeBSD Foundation
2020-03-17 14:12:19 +00:00
Ed Maste
450cc34172 Makefile.inc1: move dependency hack comment to the block it applies to 2020-03-12 14:01:17 +00:00
Ed Maste
3b1781b4fd Extend r358907 to explicitly remove stale lib32 dependency
After r325072 stale lib32 dependencies were not remooved.  A more
holistic approach is needed to address this but for the immediate issue
(-DNO_CLEAN builds across r358851) just readd the explicit lib32 path.

Reported by:	dim
Sponsored by:	The FreeBSD Foundation
2020-03-12 13:42:08 +00:00
Dimitry Andric
0efe25f9a5 Allow -DNO_CLEAN build across r358851.
The openmp 10.0.0 import renamed one .c file to .cpp, and this is
something our dependency system does not handle correctly.  Add another
ad-hoc cleanup to get rid of the stale dependency.

PR:		244251
MFC after:	6 weeks
X-MFC-With:	358851
2020-03-12 11:39:04 +00:00
Ed Maste
13f7dbe822 retire amd(8)
autofs was introduced with FreeBSD 10.1 and is the supported method for
automounting filesystems.  As of r296194 the amd man page claimed that it
is deprecated.  Remove it from base now; the sysutils/am-utils port is
still available if necessary.

Discussed with:	cy
Relnotes:	Yes
Sponsored by:	The FreeBSD Foundation
2020-03-09 20:46:43 +00:00
Ed Maste
134b378392 retire in-tree GPL dtc devicetree compiler
Now that we no longer have GCC 4.2.1 in the tree and can assume FreeBSD
is being built with a C++11 compiler available, we can use BSDL dtc
unconditionally and retire the GPL dtc.

GPL dtc now has FreeBSD CI support via Cirrus-CI to help ensure it
continues to build/work on FreeBSD and is available in the ports tree
if needed.

The copy of (copyfree licensed) libfdt that we actually use is in
sys/contrib/libfdt so the extra copy under contrib/dtc/libfdt can be
removed along with the rest of the GPL dtc.

Reviewed by:	kevans, ian, imp, manu, theraven
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D23192
2020-02-29 17:10:54 +00:00
Ed Maste
c45018041d retire the LLVM_LIBUNWIND option
LLVM's libunwind is used on all FreeBSD-supported CPU architectures and
is a required component.

Reviewed by:	brooks (earlier)
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D23123
2020-02-29 12:43:43 +00:00
Ed Maste
57f804675e remove GCC 4.2.1 build infrastructure
As described in Warner's email message[1] to the FreeBSD-arch mailing
list we have reached GCC 4.2.1's retirement date.  At this time all
supported architectures either use in-tree Clang, or rely on external
toolchain (i.e., a contemporary GCC version from ports).

GCC 4.2.1 was released July 18, 2007 and was imported into FreeBSD later
that year, in r171825.  GCC has served us well, but version 4.2.1 is
obsolete and not used by default on any architecture in FreeBSD.  It
does not support modern C and does not support arm64 or RISC-V.

Thanks to everyone responsible for maintaining, updating, and testing
GCC in the FreeBSD base system over the years.

So long, and thanks for all the fish.

[1] https://lists.freebsd.org/pipermail/freebsd-arch/2020-January/019823.html

PR:		228919
Reviewed by:	brooks, imp
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D23124
2020-02-29 03:25:51 +00:00
Dimitry Andric
eb834d9f1b Take LINKER_FREEBSD_VERSION from numerical field after dash
Summary:
With COMPILER_FREEBSD_VERSION, we use a numeric value that we bump each
time we make a change that requires re-bootstrapping, but with the
linker variant, we instead take the entire part after "FreeBSD", as in
this example version output:

LLD 9.0.1 (FreeBSD c1a0a213378a458fbea1a5c77b315c7dce08fd05-1300006) (compatible with GNU linkers)

E.g., LINKER_FREEBSD_VERSION is currently being set to
"c1a0a213378a458fbea1a5c77b315c7dce08fd05-1300006".  This means that
*any* new upstream lld version will cause re-bootstrapping.

We should only look at the numerical field we append after a dash
instead.  This review attempts to make it so.

The only thing I am not happy about is the post-processing of awk output
in Makefile.inc1.  I notice that our awk does not have gensub(), so it
can't substitute a numbered sub-regex with \1, \2, etc.  Suggestions
welcome. :)

MFC after:	1 week
Differential Revision: https://reviews.freebsd.org/D23691
2020-02-19 21:12:59 +00:00
Warner Losh
75ababf8b5 Restore missing comment
I was overly agressive about removing the entire comment. It was still
valid, except the part about being only for some architectures.

Reviewed by: emaste
Differenial Revision: https://reviews.freebsd.org/D23523
2020-02-06 17:51:48 +00:00
Warner Losh
5672c76477 Per the firm plan, start to remove sparc64
The sparc64 architecture is being removed from FreeBSD 13, starting
now. This removes it from the top level only. It is the only
architecture that didn't see substantial work after the call to get
things working with the external toolchain.
2020-02-02 11:37:27 +00:00
Ed Maste
43e8403953 retire BSD_CRTBEGIN option
BSD crt is currently used on all architectures (other than sparc64).
Remove the option and use BSD crt everywhere as part of the GCC 4.2.1
retirement plan.

https://lists.freebsd.org/pipermail/freebsd-arch/2020-January/019823.html

PR:		239851
Reviewed by:	andrew, brooks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D23122
2020-01-31 18:04:04 +00:00
John Baldwin
7ec5e1c4cd Remove support for auto-selecting an external binutils.
All of the in-tree architectures not supported by in-tree binutils are
supported by lld, so the condition is now always false.  It also
didn't fully work since the external binutils are installed into a
directory that uses the host's OS version, not the target OS version.

Reviewed by:	emaste, imp
Differential Revision:	https://reviews.freebsd.org/D23294
2020-01-22 21:21:24 +00:00
Ed Maste
9ccace3f7c pkgbase: allow the pkg format to be overridden
Compressing .txz packages can be rather slow, and speed is likely more
important than disk space during development.  Allow package format to
be set via PKG_FORMAT make variable.

Reviewed by:	bapt
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D23257
2020-01-19 14:46:28 +00:00
Ed Maste
c530c7915b pkgbase: cache pkg ABI for all world/kernel packages
Rather than invoking `pkg config ABI` repeatedly.

Reviewed by:	manu
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D23255
2020-01-18 22:58:32 +00:00
Kyle Evans
4f47920e9c Makefile.inc1: push /usr/libexec into the BPATH/TMPPATH
${WORLDTMP}/legacy/usr/libexec will only have libexec/ bits that we've
pushed as bootstrap tools, so this is generally safe to include prior to
PATH. The following are the ramifications of this change:

- BPATH addition gets us at least bootstrap flua in WMAKEENV path for
  buildenv, for those earlier systems where it's bootstrapped still

- Reworked the sysent target to just set PATH and let it get worked out in
  src.lua.mk or individual sysent makefiles -- this gives us back the
  ability to overwrite LUA_CMD and use a different/external lua for these
  targets.  sysent can also now work cleanly in buildenv.

- tools/build/Makefile will now symlink the host flua into build's host
  tools so that the above can work without needing to add the host's
  /usr/libexec explicitly into TMPPATH.

Reviewed by:	arichardson, brooks, imp (all slightly earlier version)
Differential Revision:	https://reviews.freebsd.org/D22464
2020-01-12 04:18:36 +00:00