This is mostly targetting stable/10 which requires bmake to build and
has issues upgrading from <10.
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
PR: 198062
This will ensure that the variable was not set as a make override, in
make.conf, src.conf or src-env.conf. It allows setting the value in
src-env.conf when using WITH_AUTO_OBJ since that case properly handles
changing .OBJDIR (except if MAKEOBJDIRPREFIX does not yet exist which is
being discussed to be changed).
This change allows setting a default MAKEOBJDIRPREFIX via local.sys.env.mk.
Sponsored by: EMC / Isilon Storage Division
RISC-V is a new ISA designed to support computer research and education, and
is now become a standard open architecture for industry implementations.
This is a minimal set of changes required to run 'make kernel-toolchain'
using external (GNU) toolchain.
The FreeBSD/RISC-V project home: https://wiki.freebsd.org/riscv.
Reviewed by: andrew, bdrewery, emaste, imp
Sponsored by: DARPA, AFRL
Sponsored by: HEIF5
Differential Revision: https://reviews.freebsd.org/D4445
Say it with me, "I will not chain commands with && in Makefiles"
This was originally fixed and explained quite well by bde@ in r36074. The
initial bmake integration caused 'set -e' to stop being used which lead to
r252419. Later 'set -e' expectations were fixed with bmake in r254980.
Because of the && here, errors would be ignored when building in parallel and
a dependency failed. Such as bootstrap-tools since it builds everything in
parallel. If any tool failed in obj/depend/all, it would just ignore the error
and continue to build. This later would result in cascaded errors that only
confused the real issue. This could also cause commands after the failed
command to still execute, leading to more confusion.
This should be fine if the command is in a sub-shell such as: (cmd1 && cmd2)
This reverts r252419.
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
This allows META_FILES option to be renamed META_MODE.
Also add META_COOKIE_TOUCH for use in targets that can benefit
from a cookie when in meta mode.
Differential Revision: https://reviews.freebsd.org/D4153
Reviewed by: bdrewery
This leads the way for fixing cross-build cleanup, and eventually replacing
'cleandir' with it during the build.
Sponsored by: EMC / Isilon Storage Division
MFC after: 2 weeks
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build. This is explained in
more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times. When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'. Stats are provided at the end of this message.
This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.
The option name was picked to match the existing option in ports. This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.
Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead. ccache does nothing special in these modes, although there is
no harm in calling it for them.
CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss. For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct. Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.
The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it. Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings. GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.
Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual. Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization. There is also a memcached backend available [5]. Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout. A
possible working layout is as follows:
Source: /some/prefix/src1
Source: /some/prefix/src2
Source: /some/prefix/src3
Objdir: /some/prefix/obj
Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative. Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.
distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.
The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).
Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.
The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files and ccache objects.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16
The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.
buildworld:
x buildworld-before
+ buildworld-ccache-empty
* buildworld-ccache-full
% buildworld-ccache-full-fastdep
# buildworld-fastdep
+-------------------------------------------------------------------------------+
|% * # +|
|% * # +|
|% * # xxx +|
| |A |
| A|
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 4519 4525.04 4520.73 4521.59 3.1104823
Difference at 95.0% confidence
758.027 +/- 43.4565
20.1412% +/- 1.15466%
(Student's t, pooled s = 19.1726)
* 3 1823.08 1827.2 1825.62 1825.3 2.0785572
Difference at 95.0% confidence
-1938.26 +/- 43.298
-51.5007% +/- 1.15045%
(Student's t, pooled s = 19.1026)
% 3 1266.96 1279.37 1270.47 1272.2667 6.3971113
Difference at 95.0% confidence
-2491.3 +/- 44.3704
-66.1952% +/- 1.17895%
(Student's t, pooled s = 19.5758)
# 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)
buildkernel:
x buildkernel-before
+ buildkernel-ccache-empty
* buildkernel-ccache-empty-fastdep
% buildkernel-ccache-full
# buildkernel-ccache-full-fastdep
@ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|# @ % * |
|# @ % * x + |
|# @ % * xx ++|
| MA |
| MA|
| A |
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 727.97 731.91 728.06 729.31333 2.2492295
Difference at 95.0% confidence
156.88 +/- 4.17129
27.4058% +/- 0.728695%
(Student's t, pooled s = 1.84034)
* 3 527.1 528.29 528.08 527.82333 0.63516402
Difference at 95.0% confidence
-44.61 +/- 2.33254
-7.79305% +/- 0.407478%
(Student's t, pooled s = 1.02909)
% 3 400.4 401.05 400.62 400.69 0.3306055
Difference at 95.0% confidence
-171.743 +/- 2.16453
-30.0023% +/- 0.378128%
(Student's t, pooled s = 0.954969)
# 3 201.94 203.34 202.28 202.52 0.73020545
Difference at 95.0% confidence
-369.913 +/- 2.40293
-64.6212% +/- 0.419774%
(Student's t, pooled s = 1.06015)
@ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)
[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30
PR: 182944 [4]
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
Relnotes: yes
Bmake has a documented feature of '-N' to skip executing commands which is
specifically intended for debugging top-level builds and not recursing into
sub-directories. This matches the older 'make -n' behavior we added which made
'-n -n' the recursing target and '-n' a non-recursing target.
Removing the '-n -n' feature allows the build to work as documented in
the bmake manpage with '-n' and '-N'. The older '-n -n' feature was also
not documented anywhere that I could see.
Note that the ${_+_} var is still needed as currently bmake incorrectly
executes '+' commands when '-N' is specified.
The '-n' and '-n -n' features were broken for several reasons prior to this.
r251748 made '_+_' never expand with '-n -n' which resulted in many
sub-directories not being visited until fixed 2 years later in r288391, and
many targets were given .MAKE over the past few years which resulted in
non-sub-make commands, such as rm and ln and mtree, to be executed.
This should also allow removing some indirection hacks in bsd.subdir.mk and
other cases of .USE that have a .MAKE by using '+'.
Sponsored by: EMC / Isilon Storage Division
Discussed on: arch@ (mostly silence)
The condition used matches the condition in sys.mk for setting _+_ to blank
or +.
With this -n will continue to not descend into Makefile.inc1, while -n -n will
and cause Makefile.inc1's target to run with -n.
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
This is relevant for makeman using the 'make targets' output in src.conf(5).
This makes a _UNIVERSE_TARGETS that removes arm64 if the build
requirements are not met.
bmake specific constructs not needed for make bootstrap so fmake
doesn't see them. This works with fmake just well enough for us to
build bmake to build the rest of the tree without fatal errors. Tested
only with fmake package.
arm64 relies on an external binutils port or package right now, because
the in-tree linker from binutils 2.17.50 does not support arm64. Add
arm64 to universe if the linker is available. If not output a message
that arm64 is skipped.
buildworld and buildkernel use the external binutils automatically, so
it's sufficient to run 'pkg install aarch64-binutils' to build
FreeBSD/arm64.
Differential Revision: https://reviews.freebsd.org/D2302
Reviewed by: andrew, imp
Sponsored by: The FreeBSD Foundation
only adds support for kernel-toolchain, however it is expected further
changes to add kernel and userland support will be committed as they are
reviewed.
As our copy of binutils is too old the devel/aarch64-binutils port needs
to be installed to pull in a linker.
To build either TARGET needs to be set to arm64, or TARGET_ARCH set to
aarch64. The latter is set so uname -p will return aarch64 as existing
third party software expects this.
Differential Revision: https://reviews.freebsd.org/D2005
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Dynamically figure out the list of targets based on tags passed on the mtrees
First sanity check that all packages have existing manifests
Generate the packages
Please note that for now the mtree needs more work as it has duplicate entries,
everything is not yet tagged
The packages now have generic entries and needs to be customize
building toolchain for the host computer. This toolchain produces
TARGET_ARCH and assumes the rest of the system contains libraries for
the target. It is intended to be used in a "qemu-user jail" where all
the binaries would otherwise be the target architecture's to build
ports. However, emulation of the compilers is too slow, so we build
native binaries for that. Rather than use the xdev produced binaries,
with all their weird links and paths, these binaries use the native
paths. They will not work unless installed into the qemu-user jail.
Differential Revision: https://phabric.freebsd.org/D518
Reviewed by: sbruno@
During "make buildworld", building bmake is (one of) the very first steps
and we should not be building any of its tests. Conceptually, this is the
right thing to do 1) for build simplicity reasons and 2) because there is
no need to build any tests this early on.
In practice, this fixes tinderbox builds of CURRENT from 9.x when MK_TESTS
is enabled. This is because bsd.test.mk needs some modern bmake features
not present in 9.x (:tW) and tinderbox is forcing the build to use the
CURRENT share/mk files from the very beginning (see r266617). By skipping
the build of the tests when still using the host make, we omit the problem.
Arguably, what tinderbox is doing is wrong and needs to be addressed, but
that is a separate issue.
UPDATING. This is the first step towards the removal of ia64 from
head. A buildworld for ia64 will now yield:
% make buildworld
make[1]: "/usr/src/Makefile.inc1" line 151: Unknown target ia64:ia64.
While here, trim the ia64-specific additions from ObsoleteFiles.inc
Discussed at: BSDcan
in them. This is often the case, so just ignore the return
code. Actual errors that are found will also be detected downstream in
the rare cases where the return code is 2 instead of 1.
environments (that I can't reproduce locally, but that others have
reported) seem to get tripped up by this man page install. There's
really no need to do it, so turn off the man pages using the most
portable method. We can't just directly set MK_MAN=no here because
we're bootstrapping in the host environment and such a setting was
forbidden until very recently. NO_MAN= can produce a warning, but for
now the warning is benign.
build world, so it is the only make we build or install. fmake is
still in the tree, but disconnected, and upgrades from older systems
that still have bmake has not been removed, but its state has not been
tested (it should work given how minimal the work to upgrade to bmake
is).
#NO_UNIVERSE. Many of these config files are important examples, but
add little to no regresive value to the intended purpose of
UNIVERSE. We now build over 120 kernels during universe. There's
really little to no value to this over building say 60 or even 30 of
them (either is still a way too big number). This is especially true
for kernels that are nothing more than including a common base and
adding a static DTB file. Start by pruning 1/3 of the arm kernels that
add little regresion value.
commit 1d1b908107255ffdff4d17f015d8f057d73cc6cb
Author: Brooks Davis <brooks@one-eyed-alien.net>
Date: Fri Mar 28 16:24:45 2014 +0000
Add a long needed seatbelt.
Exit with an error when make is called without a target at the top level
rather than poluting the source tree and causing use confusion in future
builds.
commit a9d9aa341b2f4308a227ab460ba85f1f287ad028
Author: Brooks Davis <brooks@one-eyed-alien.net>
Date: Tue Apr 29 16:06:12 2014 +0000
Simplify seatbelt added in 1d1b908 based in feedback.
Discussed with: imp@FreeBSD.org
Reviewed by: imp
Sponsored by: DARPA, AFRL
This targets the existing ARMv6 and ARMv7 SoCs that contain a VFP unit.
This is an optional coprocessors may not be present in all devices, however
it appears to be in all current SoCs we support.
armv6hf targets the VFP variant of the ARM EABI and our copy of gcc is too
old to support this. Because of this there are a number of WITH/WITHOUT
options that are unsupported and must be left as the default value. The
options and their required value are:
* WITH_ARM_EABI
* WITHOUT_GCC
* WITHOUT_GNUCXX
In addition, without an external toolchain, the following need to be left
as their default:
* WITH_CLANG
* WITH_CLANG_IS_CC
As there is a different method of passing float and double values to
functions the ABI is incompatible with existing armv6 binaries. To use
this a full rebuild of world is required. Because no floating point values
are passed into the kernel an armv6 kernel with VFP enabled will work with
an armv6hf userland and vice versa.
broken. None of our kernels can boot armv6eb. The little-endian kernels do
not have the required code to be able to switch endian when running a
big-endian executable.
Approved by: re (gjb)
Include PROGNAME and DESTDIR in ${MMAKE} so that it doesn't need to be
passed to each make invocation.
Suggested by: hrs
Reviewed by: hrs
Approved by: re (gjb)
A HEAD buildworld on 9.x first bootstraps bmake, but this failed when
building with standalone debug. Pass in the PROGNAME override to the
'make all' stage as well as 'make install' so that the .debug file is
created with the correct name.
Reviewed by: sjg
Sponsored by: The FreeBSD Foundation
Approved by: re (rodrigc)
it gets built 16 times in parallel in the same location.
While we are at it, until we finish getting rid of fmake,
be explicit about the make we want to use, thus avoid the problem
of the temp make being the wrong version.
Reviewed by: obrien
so that job token pipe is passed to them.
To avoid surprising anyone, only add .MAKE to ${TGTS} when -n
has not been specified (at least for Makefile).
Reviewed by: obrien
make before starting the universe targets themselves. Otherwise, all of
the targets would attempt to build make simultaneously, overwriting each
other's copies of the make object files and executable. This could lead
to strange errors, for example when partially-written make executables
are invoked.
Also amend r216620, to make the rest of universe wait properly until the
upgrade_checks target is finished, by adding universe_${target}_prologue
to the .ORDER target. Otherwise, make will be too smart for its own
good, and start building the universe targets simultaneously with the
prologues anyway.
MFC after: 1 week
starting the kernels. Before this the kernels would be built as part of the
last architecture universe target. There can cause problems when this world
finishes before the other worlds as the host compiler may be picked up
rather than the target compiler.
The solution is to add a target to build the universe kernels that depends
on all the world targets finishing. As we may not be building a world only
depend on it when MAKE_JUST_KERNELS is undefined.
The automation can set TARGET_ARCH and TARGET and then make various
top-level targets, including buildLINT and buildkernel (with
KERNCONF=LINT). Previously there was no way to generate the LINT
kernel configuration without having to do something exceptionally
painful.
1. Don't do upgrade_checks when using bmake. As long as we have WITH_BMAKE,
there's a bootstrap complication in ths respect. Avoid it. Make the
necessary changes to have upgrade_checks work wth bmake anyway.
2. Remove the use of -E. It's not needed in our build because we use ?= for
the respective variables, which means that we'll take the environment
value (if any) anyway.
3. Properly declare phony targets as phony as bmake is a lot smarter (and
thus agressive) about build avoidance.
4. Make sure CLEANFILES is complete and use it on .NOPATH. bmake is a lot
smarter about build avoidance and should not find files we generate in
the source tree. We should not have files in the repository we want to
generate, but this is an easier way to cross this hurdle.
5. Have behavior under bmake the same as it is under make with respect to
halting when sub-commands fail. Add "set -e" to compound commands so
that bmake is informed when sub-commands fail.
6. Make sure crunchgen uses the same make as the rest of the build. This
is important when the make utility isn't called make (but bmake for
example).
7. While here, add support for using MAKEOBJDIR to set the object tree
location. It's the second alternative bmake looks for when determining
the actual object directory (= .OBJDIR).
Submitted by: Simon Gerraty <sjg@juniper.net>
Submitted by: John Van Horne <jvanhorne@juniper.net>
is something for make(1) to consume. Bmake gives output such as:
"warning: Couldn't read shell's output for "/bin/sh -c true"
Note we parted from traditional Pmake behavior in r18864 / r18255.
r238211:
Support TARGET_ARCH=armv6 and TARGET_ARCH=armv6eb
This adds a new TARGET_ARCH for building on ARM
processors that support the ARMv6K multiprocessor
extensions. In particular, these processors have
better support for TLS and mutex operations.
This mostly touches a lot of Makefiles to extend
existing patterns for inferring CPUARCH from ARCH.
It also configures:
* GCC to default to arm1176jz-s
* GCC to predefine __FreeBSD_ARCH_armv6__
* gas to default to ARM_ARCH_V6K
* uname -p to return 'armv6'
* make so that MACHINE_ARCH defaults to 'armv6'
It also changes a number of headers to use
the compiler __ARM_ARCH_XXX__ macros to configure
processor-specific support routines.
Submitted by: Tim Kientzle <kientzle@freebsd.org>
This makes our naming scheme more closely match other systems and the
expectations of much third-party software. MIPS builds which are little-endian
should require and exhibit no changes. Big-endian TARGET_ARCHes must be
changed:
From: To:
mipseb mips
mipsn32eb mipsn32
mips64eb mips64
An entry has been added to UPDATING and some foot-shooting protection (complete
with warnings which should become errors in the near future) to the top-level
base system Makefile.
