Inheriting $PATH during the build phase can cause the build to fail when
compiling on a different system due to missing build tools or incompatible
versions somewhere in $PATH. This has cause build failures for us before
due to the jenkins slaves still running FreeBSD 10.
Listing the tools we depend on explicitly instead of just using whatever
happens to be in $PATH allows us to check that we don't accidentally add a
new build dependency.
All tools that do no need to be bootstrapped will now be symlinked to
${WORLDTMP}/legacy/bin and during the build phase $PATH will only contain
${WORLDTMP}. There is also a new variable "BOOTSTRAP_ALL_TOOLS" which can
be set to force compiling almost all bootstrap tools instead of symlinking
them. This will not bootstrap tools such as cp,mv, etc. since they may be
used during the build and for those we should really only be using POSIX
compatible options.
Furthermore, this change is required in order to be able to build on
non-FreeBSD hosts. While the same binaries may exist on Linux/MacOS they
often accept different flags or produce incompatible output.
Approved By: brooks (mentor)
Differential Revision: https://reviews.freebsd.org/D16815
Further to r339946 skip the .error case for lack of in-tree binutils
support when we haven't set the toolchain feature variables.
Discussed with: arichardson
Sponsored by: The FreeBSD Foundation
This will no work when there is no cc in $PATH (which is the case before the
cross-tools stage once we no longer inherit $PATH in $WMAKE).
The variables set by bsd.compiler.mk/bsd.linker.mk are not needed in these
stages so this avoids a little bit of makefile parsing.
Reviewed By: emaste
Approved By: jhb (mentor)
Differential Revision: https://reviews.freebsd.org/D16814
It is more convenient to review old libraries, files, and directories
in order.
Sort check-* after checking for existence of files for efficiency, and
because /usr/lib/debug entries are added while iterating over the list.
Reviewed by: brooks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17649
When building CheriBSD we have to set XLD/XCC/XCFLAGS on the command line.
This triggers the $XCC != $CC case in bsd.compiler.mk (and the same for LD
in bsd.linker.mk) which causes it to call ${XCC} --version and
${XLD} --version (plus various awk+sed+echo calls) in every subdirectory.
For incremental builds and stages that only walk the source tree this is
often the majority of the time spent in that directory.
By only computing the value of the X_COMPILER_*/X_LINKER_* variables if
_WANT_TOOLCHAIN_CROSS_VARS is set we can reduce the number of cc/ld calls
to once per build stage instead of once per recursive make.
With this change (and no changes to the sources) the `make includes` stage
now takes 28 seconds at -j1 instead of 86 seconds.
Approved By: brooks (mentor)
Differential Revision: https://reviews.freebsd.org/D17046
Our dependency tracking cannot directly cope with certain source tree
changes, particularly with respect to removing or moving source files or
replacing generated files. We have a collection of ad-hoc workarounds
to handle these cases. As there is a (small) build-time cost inherent
in these workarounds, we do not want to keep them indefinitely. Thus,
remove workarounds from 2017.
Sponsored by: The FreeBSD Foundation
r339348 switched bcopy from .s to .c. Add ad-hoc dependency cleanup
as done for similar cases.
Approved by: re (kib)
Sponsored by: The FreeBSD Foundation
For OpenSSL 1.1.1 compatibility.
In Makefile.inc1 add (to the existing similar cases) a hack to handle
dependencies across the migration.
Reviewed by: jhb
Approved by: re (gjb)
Sponsored by: The FreeBSD Foundation.
Differential Revision: https://reviews.freebsd.org/D17481
Update the BOOTSTRAPPING check for libelf to require the fix for
mips64el object files committed in r338478 and re-enable kernel
modules in the MALTA64EL config file.
Reviewed by: emaste
Approved by: re (gjb)
Differential Revision: https://reviews.freebsd.org/D17054
During the build it is only used by share/ctypedef and share/colldef
which will not be built if ${MK_LOCALE} == "no". This saves a tiny bit
of time when building without locales.
Approved By: jhb (mentor)
This has two advantages:
1) We no longer create lots of empty directories that are not needed
2) This is a requirement for building on non-FreeBSD hosts since mtree will
only exist after the bootstrap-tools phase there.
Aproved By: jhb (mentor)
Differential Revision: https://reviews.freebsd.org/D16773
Some background: in the GSoC project, libbe/Makefile lived in lib/libbe. I
created projects/bectl branch, maintained the above for all of five
minutes before I misread Makefile.inc1 and decided that it couldn't possibly
build outside of cddl/, so I kicked the Makefile out into the cddl/ build
and all was good. The misreading was of the bit where .WAIT is added to
SUBDIR after lib, libexec but prior to building bin and cddl *only during
the install targets*, which is the critical part.
Fast forward- buildworld was still broken in my branch unbeknownst to me
because I didn't nuke my OBJDIR. Combing through Makefile.inc1 eventually
revealed the necessary magic to make sure that libbe's dependencies are
specified well enough, and it becomes clear what needs done to make a
non-cddl/ build work. This is an interesting prospect, because the build
split is kind of annoying to work with.
IGNORE_PRAGMA is added to avoid dropping WARNS by one more. This was
previously pulled in via cddl/Makefile.inc.
In the create-world-packages target we manually piece this together (unless
it is undefined), without the DISTDIR. Normally DISTDIR is empty (unset) and
no one notices. Now DISTDIR is a well known long-standing PORTS environment
variable and if that is set in the local environment the path to METALOG
is wrong as it no longer is ${DESTDIR}/METALOG.
Long-term we should start to avoid "publicly well known" names for global
variables, for now just piece ${DISTDIR} in as well. This allows
create-world-packages to continue if DISTDIR is set in the env.
There are several scripts and targets solely used to generate install
media, make sure DB_FROM_SRC is used in that case in order to prevent
checking the host database, which is irrelevant when generating
install binaries.
Sponsored by: Citrix Systems R&D
PR: 230459
Reviewed by: gjb
Differential revision: https://reviews.freebsd.org/D16638
- Missing include path
- Fully specify libzfs's dependencies (except for deps pulled in by other
deps) in Makefile.inc1
- Drop WARNS back down to 2 for libbe(3). I do this with much hesitation,
but the libzfs headers are apparently a hot warning-filled mess as far as
GCC 4.2 is concerned.
libl is needed for config(8), which is a bootstrap-tool. It is possible to
build a system WITHOUT_TOOLCHAIN to exclude lex and thus, libl. We still
need to support building from this kind of host, though.
While here, group the config(8) dependencies together and add a small
explanation. These can likely both be scoped more clearly, but this will
need some further investigation.
Reported by: rgrimes (not WITHOUT_TOOLCHAIN, but provoked investigation)
MFC after: immediately
distributeworld is used to generate install media, so it makes no
sense to check the host database since the install media can be
generated from any box, regardless of the version of FreeBSD it's
running.
Sponsored by: Citrix Systems R&D
Reviewed by: ian, gjb
Differential revision: https://reviews.freebsd.org/D16507
r335653 flipped the order in which hints/env files are concatenated to match
the order in which vars are processed by the kernel. This is the other
hammer to drop.
Use nv(9) to de-dupe entries within a single `hint` or `env` file, using the
latest value specified for a key. This leaves some duplicates if a variable
is specified in multiple hint/env files or via `envvar` in a kernel config,
but the reversed order of concatenation (from r335653) makes this a
non-issue as the latest-specified version will be seen first.
This change also silently rewrote hint bits to use the same sanitization
process that ian@ wrote for r335642. To the kernel, hints and env vars are
basically the same thing through early boot, then get merged into the
dynamic environment once kmem becomes available and the dynamic environment
is created. They should be subjected to the same restrictions.
libnv has been added to -legacy for the time being to support the build of
config(8) with the new cnvlist API.
Tested with: universe (11 host & 12 host)
MFC after: 1 month
The build of this would have MK_LLVM_TARGET_ALL=yes for bootstrap-tools
but MK_LLVM_TARGET_ALL=no for cross-tools and thus would rebuild between
the two.
MFC after: 2 weeks
X-MFC-with: r335707
Sponsored by: Dell EMC
Currently the mtree calls in Makefile.inc1 all change the directory owner
to match the spec file. However, we should not be doing this during
distributeworld if -DNO_ROOT is passed. Additionally, when creating the
WORLDTMP directory hierachy there is no need to change the owner to root so
we now always pass the -W flag when populating WORLDTMP.
This is also required for building FreeBSD on Linux/Mac since the required
groups/users will not exist there which is how I discovered this issue.
Reviewed By: emaste, bdrewery, imp
Approved By: brooks (mentor)
Differential Revision: https://reviews.freebsd.org/D14185
The xtoolchain GCC packages have not required these flags since ports
commits r465416 and r466701. The in-tree GCC 4.2.1 has also been patched
in r335716 and r335717 to correctly honor --sysroot when looking for
includes and libraries.
Reviewed by: bdrewery
Sponsored by: DARPA / AFRL
Differential Revision: https://reviews.freebsd.org/D16055
Need to handle LLD_BOOTSTRAP separately (for archs like i386).
This would be much better off with an off-by-default option like
SHARED_TOOLCHAIN that universe force-enabled. Then a normal buildworld
would store the toolchain there if enabled and otherwise in WORLDTMP
with only the 1 arch selected.
MFC after: 3 weeks
Sponsored by: Dell EMC
This will disable the new LLVM_TARGET_ALL option which will only
enable the required target.
This only impacts the bootstrap compiler in WORLDTMP, not the target compiler
that will be installed.
MFC after: 2 weeks
Reviewed by: sbruno, dim (earlier version)
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D16021
LLVM_TARGET_* will auto be set based on LLVM_TARGET_ALL and MK_CLANG.
If LLVM_TARGET_ALL is disabled, during a cross-build, then SYSTEM_COMPILER
and SYSTEM_LINKER are auto disabled.
This option should be used by users rather than the per-arch LLVM_TARGET
options as it is simpler to maintain for them should the supported
target list change.
MFC after: 2 weeks
Reviewed by: sbruno, dim
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D16020
Rather then combining hardlink creation for the geom(8) binary with
shared library build, move libraries to src/lib/geom so they are
built and installed normally. Create a common Makefile.classes
which is included by both lib/geom/Makefile and sbin/geom/Makefile
so the symlink and libraries stay in sync.
The relocation of libraries allows libraries to be build for 32-bit
compat. This also reduces the number of non-standard builds in
the system.
This commit is not sufficent to run a 32-bit /sbin/geom on a 64-bit
system out of the box as it will look in the wrong place for libraries
unless GEOM_LIBRARY_PATH is set appropriatly in the environment.
Reviewed by: bdrewery
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D15360
We always install ELF Tool Chain's elfcopy as objcopy, so to avoid
confusion rename the src directory containing our reach-over Makefile
to match.
Requested by: jhb
Sponsored by: The FreeBSD Foundation
This works similar to WITH_SYSTEM_COMPILER added in r300354. It only
supports lld via WITH_LLD_BOOTSTRAP.
When both SYSTEM_COMPILER and SYSTEM_LINKER logic passes then libclang
will not build in cross-tools. If either check fails though then
libclang is built.
The .info is reworked to notify when libclang will be built since if
either clang or lld needs to be rebuilt, but not the other, the
notification can lead to confusion on why "clang is building".
-fuse-ld= is not used with this method so some combinations of compiler
and linker are expected to fail.
A new 'make test-system-linker' target is added to see the logic results.
Makefile.inc1:
CROSS_BINUTILS_PREFIX support had to be moved higher up so that XLD
could be set and MK_LLD_BOOTSTRAP disabled before checking SYSTEM_LINKER
logic as done with SYSTEM_COMPILER. This also required moving where
bsd.linker.mk was read since XLD needs to be set before parsing it. This
creates a situation where src.opts.mk can not test LINKER_FEATURES or
add LLD_BOOTSTAP to BROKEN_OPTIONS.
Reviewed by: emaste (earlier version)
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D15894
MK_LLD is for the installed lld while MK_LLD_BOOTSTRAP is for the build
tool. For WITH_SYSTEM_LINKER it is necesarry to separate the logic of
these two. When building libllvm TOOLS_PREFIX will be defined and
MK_LLD_BOOTSTRAP should be checked instead.
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D15837
LLD_BOOTSTRAP (build) is independent of LLD_IS_LD (installed) so they
should not be based on each other.
This is related to upcoming WITH_SYSTEM_LINKER work.
Reviewed by: emaste
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D15836
It is XCC used during the build of target binaries that replaces the
bootstrap compiler.
Also slightly tweak style.
MFC after: 2 weeks
Sponsored by: Dell EMC
This could lead to 'sh: head: not found' warnings which were a symptom
of running 'ld --version' during installworld. This was only happening
with XCC or XLD set. It is intended that cc and ld do not run during
installworld. The metadata for these are already stored in
compiler-metadata.mk added in r316794.
This also removes redundant CROSSENV additions that were for
WITH_SYSTEM_COMPILER, WITHOUT_CROSS_COMPILER, and WITHOUT_TOOLCHAIN
which all don't have a cc or ld in their PATH during install.
Reported by: Mark Millard
MFC after: 2 weeks
Sponsored by: Dell EMC
Aligns the build with the FreeBSD traditional approach to not build in
contrib/, and to track inter-dependencies between libraries.
With help from: bdrewery
Reviewed by: bdrewery, hselasky
Sponsored by: Mellanox Technologies
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D15648
vendor provided pmu-events tables and sundry cleanups.
The vendor pmu-events tables provide counter descriptions, default
sample rates, event, umask, and flag values for all the counter
configuration permutations. Using this gives us:
- much simpler kernel code for the MD component
- helpful long and short event descriptions
- simpler user code
- sample rates that won't overload the system
Update man page with newer sample types and remove unused sample type.
vendor provided pmu-events tables and sundry cleanups.
The vendor pmu-events tables provide counter descriptions, default
sample rates, event, umask, and flag values for all the counter
configuration permutations. Using this gives us:
- much simpler kernel code for the MD component
- helpful long and short event descriptions
- simpler user code
- sample rates that won't overload the system
Update man page with newer sample types and remove unused sample type.
Squashed commit of the following:
commit 4459d43eff815bec08ccc5533dbe5de846f03128
Author: Matt Macy <mmacy@mattmacy.io>
Date: Sat May 26 00:06:31 2018 -0700
libpmc: fix pmu function signatures for non amd64
commit a2cb8bbc586c65d41f9b291430a2261ec67b59fe
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:38:11 2018 -0700
pmcstat: fix indentation of usage
commit f686954b15ff56a833ac80404898977cb80a265b
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:19:49 2018 -0700
pmclog(3): add callchain and pmcallocatedyn, remove pcsample
commit 73e13a0d2e9498c81c150d14d022050cee7511bb
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:19:00 2018 -0700
pmclog.h: GC pcsample field
commit 3e93ffd65da641fa657539dad3c48e281f8b5798
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:05:57 2018 -0700
hwpmc: make Intel core CPUs use external event tables
commit 634f5fae1e1644ac324003136c66cd9c619d1c93
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 22:00:06 2018 -0700
pmclog: update log record types, bump PMC_MAJOR
- explicitly make log record types a multiple of 8 bytes
- hook in pmu event types for pmc_allocate records
- remove references to no longer PCSAMPLE record
commit 83d84fcd2d65bdf6ddcb2e155a22f0cfa2a9c225
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 21:52:10 2018 -0700
libpmc: add support for having vendor table driven pmc_allocate
commit 9e6ad63c40c2fce8404847ace5078ca6cb33a736
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 19:11:33 2018 -0700
hwpmc_core: add accessors for EVSEL & UMASK, make IAP_UMASK useful to user
commit 859dceb93daa6419a48c794db99b6758e5b041c9
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 19:09:45 2018 -0700
pmcstat: update usage and man page as well as make -L consistent with pmccontrol
commit 79c7d8597e28c2eb13f5f9113e65ec2792ca57b1
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 18:07:03 2018 -0700
pmu_util: add support for all current intel event keywords
commit d8089c7f6a6c8527f38324252b1ffb47004694c6
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 17:45:00 2018 -0700
add description for new arguments
commit 058336740bab53c62ec88a3a026ea848cf3878c6
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 17:38:15 2018 -0700
libpmc: move pmu_events table and pmu_utils out of libpmcstat so that they can be used by pmc_allocate
commit 049b66b382e2f833c3f47bc8df9e750cb265709f
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 16:12:41 2018 -0700
pmcstat: hook pmu_events counter description utility routines in
commit f5e01e7b37a691dc045e1aa16b3ebdd162515de8
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 16:11:59 2018 -0700
pmu_events: add utility routines for listing counters and their descriptions
commit cba4d4f8907f772279f86f18f915e0d74d33ac56
Author: Matt Macy <mmacy@mattmacy.io>
Date: Fri May 25 16:09:50 2018 -0700
pmu-events: expand out skylake regex to simplify string matches
r307825 and r333017 disallow building FreeBSD-HEAD from 9.x or 10.x
versions that have some clang issues. The minimum supported Subversion
revisions and osreldates containing the fix are:
Version Min Rev osreldate
9.x r286035 903509
10.x r286033 1002501
9.3 is the final 9.x release and does not contain the r286035 fix.
10.3 is the first 10.x release with the fix. Thus, in practice 10.3 is
the oldest release that can build HEAD.
Although it may still be possible to build HEAD from an up-to-date
stable/9 it's not worth maintaining the special case when the branch
itself is unsupported and there are no usable releases from that branch.
Old UPDATING entries can be removed and the Clang warning in UPDATING
may be updated, in a future commit.
Approved by: imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D15209
r318957 added special handling for stale ptrace dependency files to
support a -DNO_CLEAN build in a tree last built before r305012. That
revision is now over a year and a half old, so retire the special case.
Sponsored by: The FreeBSD Foundation
So that -DNO_CLEAN builds work.
This should be reverted after a sufficient transition period (perhaps 3
or 6 months).
Sponsored by: The FreeBSD Foundation
kernel.ucl uses a hardcoded boot/kernel for kldxref, which is the incorrect
directory when we're installing extra kernels that aren't the "default"
kernel (placed at boot/kernel).
Fix this by instead using a new %KERNELDIR% that we now replace in
Makefile.inc1 with "kernel" for the default kernel and "kernel.${_kernel}"
for these extra kernels so that, e.g. /boot/kernel.SHIVA, will get properly
kldxref'd upon update and avoid outdated linker.hints.
Reviewed by: gjb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D14689
This allows working with custom cross toolchains without the need to
create files in /usr/local/share/toolchains.
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D14178
This code, which is basically `svnversion || svnliteversion`, generates
2 fstatat(2) for every directory in PATH for every Makefile parsed that
includes bsd.own.mk. This can add up for things like generating a Ports
index (Poudriere) or building a dependency graph for base.
Sponsored by: Dell EMC
MFC after: 2 weeks
bsd.compiler.mk. It's so fmake from older 9.x systems still
works (still a supported build config, and having the note here
will let us know when we can cull it more easily).
Also pull in a related change from include to sinclude from
arichardson@'s cross building work, as well as it's companion in
Makefile.inc1 with a note about why we do the odd thing there.
Submitted by: archardson
Differential Revision: https://reviews.freebsd.org/D14241
These features indicate that the compiler and linker support the
retpoline speculative execution vulnerability (CVE-2017-5715)
mitigation.
Reviewed by: dim, imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D14228
It won't work e.g. when crossbuilding from Ubuntu Linux as mktemp is in
/bin there.
Reviewed By: bdrewery
Approved By: jhb (mentor)
Differential Revision: https://reviews.freebsd.org/D13937
This fixes building libllvmminimal with MK_LLD support when WITHOUT_LLD
or WITHOUT_TOOLCHAIN are set, which are install flags not build tools
flags.
Reported by: sbruno
Tested by: sbruno
Sponsored by: Dell EMC
A comment in Makefile.inc1 has long stated that LOCAL_DIRS are built last,
after the base system. Incremental improvements in parallel building over
the years have led to LOCAL_DIRS being built in parallel with base system
directories. This change allows the .WAIT directive to appear in LOCAL_DIRS
and LOCAL_LIB_DIRS lists to give the user some control over parallel
building of local additions.
Differential Revision: https://reviews.freebsd.org/D13622
