d/freebsd-dev
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
b4ef09b9eb
freebsd-dev/share/mk/bsd.compiler.mk

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

298 lines
12 KiB
Makefile
Raw Normal View History

Introduce a new make variable COMPILER_TYPE that specifies what type of compiler is being used (currently clang or gcc). COMPILER_TYPE is set in the new bsd.compiler.mk file based on the value of the CC variable or, should it prove informative, by running ${CC} --version and examining the output. To avoid negative performance impacts in the default case and correct value for COMPILER_TYPE type is determined and passed in the environment of submake instances while building world. Replace adhoc attempts at determining the compiler type by examining CC or MK_CLANG_IS_CC with checks of COMPILER_TYPE. This eliminates bootstrapping complications when first setting WITH_CLANG_IS_CC. Sponsored by: DARPA, AFRL Reviewed by: Yamaya Takashi <yamayan@kbh.biglobe.ne.jp>, imp, linimon (with some modifications post review) MFC after: 2 weeks
2012-09-13 16:00:46 +00:00
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
# Setup variables for the compiler
#
Fix typos in comments and wrap to <80 columns MFC after: 3 days
2014-12-06 03:12:57 +00:00
# COMPILER_TYPE is the major type of compiler. Currently gcc and clang support
# automatic detection. Other compiler types can be shoe-horned in, but require
# explicit setting of the compiler type. The compiler type can also be set
# explicitly if, say, you install gcc as clang...
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
#
Fix typos in comments and wrap to <80 columns MFC after: 3 days
2014-12-06 03:12:57 +00:00
# COMPILER_VERSION is a numeric constant equal to:
# major * 10000 + minor * 100 + tiny
Fix spelling error. Reported by: arc Sponsored by: Dell EMC Isilon
2017-06-22 20:32:23 +00:00
# It too can be overridden on the command line. When testing it, be sure to
Fix typos in comments and wrap to <80 columns MFC after: 3 days
2014-12-06 03:12:57 +00:00
# make sure that you are limiting the test to a specific compiler. Testing
# against 30300 for gcc likely isn't what you wanted (since versions of gcc
# prior to 4.2 likely have no prayer of working).
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
#
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
# COMPILER_FREEBSD_VERSION is the compiler's __FreeBSD_cc_version value.
#
Fix typos in comments and wrap to <80 columns MFC after: 3 days
2014-12-06 03:12:57 +00:00
# COMPILER_FEATURES will contain one or more of the following, based on
# compiler support for that feature:
#
Add c++14 and c++17 to COMPILER_FEATURES This will be used to gate the fusefs tests. It's a partial merge of r348281 from projects/fuse2. Reviewed by: kib, emaste MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21044
2019-07-24 15:10:09 +00:00
# - c++17: supports full (or nearly full) C++17 programming environment.
# - c++14: supports full (or nearly full) C++14 programming environment.
add retpoline compiler and linker feature flags These features indicate that the compiler and linker support the retpoline speculative execution vulnerability (CVE-2017-5715) mitigation. Reviewed by: dim, imp Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D14228
2018-02-07 14:50:06 +00:00
# - c++11: supports full (or nearly full) C++11 programming environment.
# - retpoline: supports the retpoline speculative execution vulnerability
# mitigation.
Support initializing stack variables on function entry There are two options: - WITH_INIT_ALL_ZERO: Zero all variables on the stack. - WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns. The exact pattern are a compiler implementation detail and vary by type. They are somewhat documented in the LLVM commit message: https://reviews.llvm.org/rL349442 I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather than naming them after the LLVM specific compiler flags. In a range of consumer products, options like these are used in both debug and production builds with debugs builds using patterns (intended to provoke crashes on use of uninitialized values) and production using zeros (deemed more likely to lead to harmless misbehavior or NULL-pointer dereferences). Reviewed by: emaste Obtained from: CheriBSD Sponsored by: DARPA Differential Revision: https://reviews.freebsd.org/D27131
2020-11-10 19:15:13 +00:00
# - init-all: supports stack variable initialization.
Add an arm64 sha512 feature check This is only available in recent toolchains. Check for it so we can use it in libmd. Reviewed by: emaste Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D33166
2021-11-29 17:21:31 +00:00
# - aarch64-sha512: supports the AArch64 sha512 intrinsic functions.
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
#
bsd.compiler.mk: detect Apple Clang for cross-builds Apple clang uses a different versioning scheme, so if we enable or disable certain warnings for Clang 11+, those might not be supported in Apple Clang 11+. This adds 'apple-clang' to COMPILER_FEATURES, so that bootstrap tools Makefiles can avoid warnings on macOS. Reviewed By: imp Differential Revision: https://reviews.freebsd.org/D29680
2021-04-20 00:46:36 +00:00
# When bootstrapping on macOS, 'apple-clang' will be set in COMPILER_FEATURES
# to differentiate Apple's version of Clang. Apple Clang uses a different
# versioning scheme and may not support the same -W/-Wno warning flags. For a
# mapping of Apple Clang versions to upstream clang versions see
# https://en.wikipedia.org/wiki/Xcode#Xcode_7.0_-_12.x_(since_Free_On-Device_Development)
#
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
# These variables with an X_ prefix will also be provided if XCC is set.
#
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
# This file may be included multiple times, but only has effect the first time.
#
bsd.compiler.mk was implicitly included by bsd.own.mk in historical versions. With its movement to src.opts.mk, bsd.prog.mk was testing COMPILER_TYPE without including the bsd.compiler.mk anymore. In the source tree, this caused no problems, for reasons that aren't clear, but does cause problems outside of the source tree. Allow bsd.compiler.mk to be included multiple times safely, and always include bsd.compiler.mk at the top of bsd.prog.mk. Resist the urge to put it in bsd.init.mk, since that would reintroduce the implicit include.
2014-05-07 18:15:02 +00:00
.if !target(__<bsd.compiler.mk>__)
__<bsd.compiler.mk>__:
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
.include <bsd.opts.mk>
Don't run cc --version during cleandir/obj stages This will no work when there is no cc in $PATH (which is the case before the cross-tools stage once we no longer inherit $PATH in $WMAKE). The variables set by bsd.compiler.mk/bsd.linker.mk are not needed in these stages so this avoids a little bit of makefile parsing. Reviewed By: emaste Approved By: jhb (mentor) Differential Revision: https://reviews.freebsd.org/D16814
2018-10-31 10:45:28 +00:00
.if defined(_NO_INCLUDE_COMPILERMK)
# If _NO_INCLUDE_COMPILERMK is set we are doing a make obj/cleandir/cleanobj
# and might not have a valid compiler in $PATH yet. In this case just set the
# variables that are expected by the other .mk files and return
COMPILER_TYPE=none
X_COMPILER_TYPE=none
COMPILER_VERSION=0
X_COMPILER_VERSION=0
COMPILER_FEATURES=none
.else
CCACHE_BUILD: Allow setting CCACHE_BUILD_TYPE=wrapper. This uses the /usr/local/libexec/ccache/<cc,c++> wrappers rather than modifying CC to be '/usr/local/bin/ccache cc'. Some forms of compilation do not support the 'command' type. Sponsored by: Dell EMC Isilon
2017-08-01 16:15:08 +00:00
# command = /usr/local/bin/ccache cc ...
# wrapper = /usr/local/libexec/ccache/cc ...
CCACHE_BUILD_TYPE?= command
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
# Handle ccache after CC is determined, but not if CC/CXX are already
# overridden with a manual setup.
.if ${MK_CCACHE_BUILD:Uno} == "yes" && \
Reduce exec and fstat overhead for non-build targets. This is mostly targetting 'make showconfig' and 'make test-system-*' for the benefit of makeman and universe-one-clang work. Sponsored by: Dell EMC
2018-06-20 17:20:39 +00:00
!make(test-system-*) && !make(print-dir) && !make(showconfig) && \
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
(${CC:M*ccache/world/*} == "" || ${CXX:M*ccache/world/*} == "")
# CC is always prepended with the ccache wrapper rather than modifying
# PATH since it is more clear that ccache is used and avoids wasting time
# for mkdep/linking/asm builds.
LOCALBASE?= /usr/local
bsd.compiler.mk: Use CCACHE_PKG_PREFIX as ports now supports.
2021-07-12 19:51:41 +00:00
CCACHE_PKG_PREFIX?= ${LOCALBASE}
CCACHE_WRAPPER_PATH?= ${CCACHE_PKG_PREFIX}/libexec/ccache
CCACHE_BIN?= ${CCACHE_PKG_PREFIX}/bin/ccache
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
.if exists(${CCACHE_BIN})
# Export to ensure sub-makes can filter it out for mkdep/linking and
# to chain down into kernel build which won't include this file.
.export CCACHE_BIN
# Expand and export some variables so they may be based on make vars.
# This allows doing something like the following in the environment:
# CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
.for var in CCACHE_LOGFILE CCACHE_BASEDIR
.if defined(${var})
${var}:= ${${var}}
.export ${var}
.endif
.endfor
# Handle bootstrapped compiler changes properly by hashing their content
# rather than checking mtime. For external compilers it should be safe
# to use the more optimal mtime check.
# XXX: CCACHE_COMPILERCHECK= string:<compiler_version, compiler_build_rev, compiler_patch_rev, compiler_default_target, compiler_default_sysroot>
.if ${CC:N${CCACHE_BIN}:[1]:M/*} == ""
CCACHE_COMPILERCHECK?= content
.else
CCACHE_COMPILERCHECK?= mtime
.endif
.export CCACHE_COMPILERCHECK
# Ensure no bogus CCACHE_PATH leaks in which might avoid the in-tree compiler.
CCACHE_BUILD: Only export CCACHE_PATH= if it was already set with a value. Older ccache don't work with an empty CCACHE_PATH value. They will error with: ccache: FATAL: Could not find compiler "cc" in PATH make: "/mnt/bdrewery/git/onefs/src/share/mk/bsd.compiler.mk" line 134: Unable to determine compiler type for /usr/local/bin/ccache cc. Consider setting COMPILER_TYPE. Sponsored by: EMC / Isilon Storage Division
2015-12-08 20:20:40 +00:00
.if !empty(CCACHE_PATH)
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
CCACHE_PATH=
.export CCACHE_PATH
CCACHE_BUILD: Only export CCACHE_PATH= if it was already set with a value. Older ccache don't work with an empty CCACHE_PATH value. They will error with: ccache: FATAL: Could not find compiler "cc" in PATH make: "/mnt/bdrewery/git/onefs/src/share/mk/bsd.compiler.mk" line 134: Unable to determine compiler type for /usr/local/bin/ccache cc. Consider setting COMPILER_TYPE. Sponsored by: EMC / Isilon Storage Division
2015-12-08 20:20:40 +00:00
.endif
CCACHE_BUILD: Allow setting CCACHE_BUILD_TYPE=wrapper. This uses the /usr/local/libexec/ccache/<cc,c++> wrappers rather than modifying CC to be '/usr/local/bin/ccache cc'. Some forms of compilation do not support the 'command' type. Sponsored by: Dell EMC Isilon
2017-08-01 16:15:08 +00:00
.if ${CCACHE_BUILD_TYPE} == "command"
# Remove ccache from the PATH to prevent double calls and wasted CPP/LD time.
PATH:= ${PATH:C,:?${CCACHE_WRAPPER_PATH}(/world)?(:$)?,,g}
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
# Override various toolchain vars.
.for var in CC CXX HOST_CC HOST_CXX
.if defined(${var}) && ${${var}:M${CCACHE_BIN}} == ""
${var}:= ${CCACHE_BIN} ${${var}}
.endif
.endfor
CCACHE_BUILD: Follow-up r321880: Fix some PATH issues with buildworld. - bsd.compiler.mk: Must ensure that the CCACHE_WRAPPER_PATH comes first in PATH. - Makefile.inc1: Must prepend the CCACHE_WRAPPER_PATH into BPATH as it overrides the PATH set in bsd.compiler.mk in sub-makes. The PATH set in bsd.compiler.mk is not exported and doing so would cause it to then override the BPATH set from environment. The only sane solution is to prepend into BPATH as needed. CCACHE_PATH could possibly be used for some of this as well. Sponsored by: Dell EMC Isilon
2017-08-01 18:26:20 +00:00
.else
# Need to ensure CCACHE_WRAPPER_PATH is the first in ${PATH}
PATH:= ${PATH:C,:?${CCACHE_WRAPPER_PATH}(/world)?(:$)?,,g}
CCACHE_BUILD: Allow setting CCACHE_BUILD_TYPE=wrapper. This uses the /usr/local/libexec/ccache/<cc,c++> wrappers rather than modifying CC to be '/usr/local/bin/ccache cc'. Some forms of compilation do not support the 'command' type. Sponsored by: Dell EMC Isilon
2017-08-01 16:15:08 +00:00
PATH:= ${CCACHE_WRAPPER_PATH}:${PATH}
CCACHE_BUILD: Follow-up r321880: Fix some PATH issues with buildworld. - bsd.compiler.mk: Must ensure that the CCACHE_WRAPPER_PATH comes first in PATH. - Makefile.inc1: Must prepend the CCACHE_WRAPPER_PATH into BPATH as it overrides the PATH set in bsd.compiler.mk in sub-makes. The PATH set in bsd.compiler.mk is not exported and doing so would cause it to then override the BPATH set from environment. The only sane solution is to prepend into BPATH as needed. CCACHE_PATH could possibly be used for some of this as well. Sponsored by: Dell EMC Isilon
2017-08-01 18:26:20 +00:00
CCACHE_WRAPPER_PATH_PFX= ${CCACHE_WRAPPER_PATH}:
CCACHE_BUILD: Allow setting CCACHE_BUILD_TYPE=wrapper. This uses the /usr/local/libexec/ccache/<cc,c++> wrappers rather than modifying CC to be '/usr/local/bin/ccache cc'. Some forms of compilation do not support the 'command' type. Sponsored by: Dell EMC Isilon
2017-08-01 16:15:08 +00:00
.endif # ${CCACHE_BUILD_TYPE} == "command"
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
# GCC does not need the CCACHE_CPP2 hack enabled by default in devel/ccache.
# The port enables it due to ccache passing preprocessed C to clang
# which fails with -Wparentheses-equality, -Wtautological-compare, and
# -Wself-assign on macro-expanded lines.
.if defined(COMPILER_TYPE) && ${COMPILER_TYPE} == "gcc"
CCACHE_NOCPP2= 1
.export CCACHE_NOCPP2
.endif
# Canonicalize CCACHE_DIR for meta mode usage.
CCACHE_BUILD+META_MODE: Lookup and canonicalize the default CCACHE_DIR. This is done to prevent not having CCACHE_DIR causing meta mode with filemon to see stat changes in the ccache dir and cause rebuilds. Sponsored by: EMC / Isilon Storage Division
2016-03-01 22:25:54 +00:00
.if !defined(CCACHE_DIR)
CCACHE_DIR!= ${CCACHE_BIN} -p | awk '$$2 == "cache_dir" {print $$4}'
.export CCACHE_DIR
.endif
.if !empty(CCACHE_DIR) && empty(.MAKE.META.IGNORE_PATHS:M${CCACHE_DIR})
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
CCACHE_DIR:= ${CCACHE_DIR:tA}
.MAKE.META.IGNORE_PATHS+= ${CCACHE_DIR}
.export CCACHE_DIR
.endif
WITH_CCACHE_BUILD + WITH_META_MODE: Ignore ccache changes. Ccache will not affect the output of the objects, so just ignore it for meta mode handling. This avoids having everything rebuild if ccache is updated. Sponsored by: EMC / Isilon Storage Division
2016-06-03 19:25:45 +00:00
# ccache doesn't affect build output so let it slide for meta mode
# comparisons.
.MAKE.META.IGNORE_PATHS+= ${CCACHE_BIN}
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
ccache-print-options: .PHONY
@${CCACHE_BIN} -p
.endif # exists(${CCACHE_BIN})
.endif # ${MK_CCACHE_BUILD} == "yes"
Only compute the X_COMPILER_*/X_LINKER_* variables when needed When building CheriBSD we have to set XLD/XCC/XCFLAGS on the command line. This triggers the $XCC != $CC case in bsd.compiler.mk (and the same for LD in bsd.linker.mk) which causes it to call ${XCC} --version and ${XLD} --version (plus various awk+sed+echo calls) in every subdirectory. For incremental builds and stages that only walk the source tree this is often the majority of the time spent in that directory. By only computing the value of the X_COMPILER_*/X_LINKER_* variables if _WANT_TOOLCHAIN_CROSS_VARS is set we can reduce the number of cc/ld calls to once per build stage instead of once per recursive make. With this change (and no changes to the sources) the `make includes` stage now takes 28 seconds at -j1 instead of 86 seconds. Approved By: brooks (mentor) Differential Revision: https://reviews.freebsd.org/D17046
2018-10-23 06:31:25 +00:00
_cc_vars=CC $${_empty_var_}
.if !empty(_WANT_TOOLCHAIN_CROSS_VARS)
# Only the toplevel makefile needs to compute the X_COMPILER_* variables.
# Skipping the computation of the unused X_COMPILER_* in the subdirectory
# makefiles can save a noticeable amount of time when walking the whole source
# tree (e.g. during make includes, etc.).
_cc_vars+=XCC X_
.endif
.for cc X_ in ${_cc_vars}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.if ${cc} == "CC" || !empty(XCC)
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
# Try to import COMPILER_TYPE and COMPILER_VERSION from parent make.
# The value is only used/exported for the same environment that impacts
# CC and COMPILER_* settings here.
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
_exported_vars= ${X_}COMPILER_TYPE ${X_}COMPILER_VERSION \
Fix compilation with upstream clang builtin headers. By using -nobuiltininc and adding the clang builtin headers resource dir to the end of the compiler header search path, we can still find headers such as immintrin.h but find the FreeBSD version of stddef.h/stdarg.h/.. first. This is a workaround until we are able to settle on and complete a plan to harmonize guard macros with LLVM. We've mostly worked out this on FreeBSD systems by removing select headers from the installed set of devel/llvm*, but that isn't a good solution for cross build. Submitted by: arichardson Obtained from: CheriBSD Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D17002
2020-04-06 23:38:46 +00:00
${X_}COMPILER_FREEBSD_VERSION ${X_}COMPILER_RESOURCE_DIR
GCC External: Revert r300886, r300904, r300917, r300918 The fix in r300873 is mostly enough. A fix for lib32 will be committed.separately.
2016-05-29 06:20:15 +00:00
${X_}_cc_hash= ${${cc}}${MACHINE}${PATH}
${X_}_cc_hash:= ${${X_}_cc_hash:hash}
Avoid recomputing COMPILER_/LINKER_ variables when set explicitly I noticed that when we build libraries for a different ABI (in CheriBSD) we were calling ${XCC}/${LD} --version for every directory. It turns out that this was caused by bsd.compat.mk explicitly setting (X_)COMPILER variables for that build stage and this stops the _can_export logic from working. To fix this, we change the check to only set _can_export=no if the variable is set and it is set to a different value than the cached value. This noticeably speeds up the tree walk while building compat libraries. During an upstream amd64 buildworld this also removes 8 --version calls. Obtained from: CheriBSD Reviewed By: brooks, emaste Differential Revision: https://reviews.freebsd.org/D25986
2020-08-26 10:21:38 +00:00
# Only import if none of the vars are set differently somehow else.
Improve safety of caching from r289659 by only importing of none of the variables are already set. This should cover odd cases such as the COMPILER_TYPE override in lib/csu/powerpc64. X-MFC-With: r289659 MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division
2015-10-20 20:37:00 +00:00
_can_export= yes
.for var in ${_exported_vars}
Avoid recomputing COMPILER_/LINKER_ variables when set explicitly I noticed that when we build libraries for a different ABI (in CheriBSD) we were calling ${XCC}/${LD} --version for every directory. It turns out that this was caused by bsd.compat.mk explicitly setting (X_)COMPILER variables for that build stage and this stops the _can_export logic from working. To fix this, we change the check to only set _can_export=no if the variable is set and it is set to a different value than the cached value. This noticeably speeds up the tree walk while building compat libraries. During an upstream amd64 buildworld this also removes 8 --version calls. Obtained from: CheriBSD Reviewed By: brooks, emaste Differential Revision: https://reviews.freebsd.org/D25986
2020-08-26 10:21:38 +00:00
.if defined(${var}) && (!defined(${var}__${${X_}_cc_hash}) || ${${var}__${${X_}_cc_hash}} != ${${var}})
.if defined(${var}__${${X_}_ld_hash})
Remove more quotes around Makefile .error/.warn/.info strings. Reviewed by: imp, emaste Differential Revision: https://reviews.freebsd.org/D34175
2022-02-04 23:59:00 +00:00
.info Cannot import ${X_}COMPILER variables since cached ${var} is different: ${${var}__${${X_}_cc_hash}} != ${${var}}
Avoid recomputing COMPILER_/LINKER_ variables when set explicitly I noticed that when we build libraries for a different ABI (in CheriBSD) we were calling ${XCC}/${LD} --version for every directory. It turns out that this was caused by bsd.compat.mk explicitly setting (X_)COMPILER variables for that build stage and this stops the _can_export logic from working. To fix this, we change the check to only set _can_export=no if the variable is set and it is set to a different value than the cached value. This noticeably speeds up the tree walk while building compat libraries. During an upstream amd64 buildworld this also removes 8 --version calls. Obtained from: CheriBSD Reviewed By: brooks, emaste Differential Revision: https://reviews.freebsd.org/D25986
2020-08-26 10:21:38 +00:00
.endif
Improve safety of caching from r289659 by only importing of none of the variables are already set. This should cover odd cases such as the COMPILER_TYPE override in lib/csu/powerpc64. X-MFC-With: r289659 MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division
2015-10-20 20:37:00 +00:00
_can_export= no
.endif
.endfor
.if ${_can_export} == yes
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
.for var in ${_exported_vars}
Use __ as the separator for the exported vars in bsd.compiler/linker.mk By using '__' instead of '.' as the separator we can also support systems that use dash as /bin/sh (it's the default shell on Ubuntu/Debian). Dash will unset any environment variables that use a non alphanumeric+undedscore character and therefore submakes will fail to import the COMPILER_* variables if we use '.' as the separator. Reviewed By: emaste Differential Revision: https://reviews.freebsd.org/D22381
2019-11-15 16:43:36 +00:00
.if defined(${var}__${${X_}_cc_hash})
${var}= ${${var}__${${X_}_cc_hash}}
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
.endif
.endfor
Improve safety of caching from r289659 by only importing of none of the variables are already set. This should cover odd cases such as the COMPILER_TYPE override in lib/csu/powerpc64. X-MFC-With: r289659 MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division
2015-10-20 20:37:00 +00:00
.endif
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.if ${cc} == "CC" || (${cc} == "XCC" && ${XCC} != ${CC})
Avoid errors when MACHINE is a pseudo machine like "common".
2013-10-12 23:49:26 +00:00
.if ${MACHINE} == "common"
Mention what 'common' is for
2015-06-10 19:36:53 +00:00
# common is a pseudo machine for architecture independent
# generated files - thus there is no compiler.
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
${X_}COMPILER_TYPE= none
${X_}COMPILER_VERSION= 0
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
${X_}COMPILER_FREEBSD_VERSION= 0
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.elif !defined(${X_}COMPILER_TYPE) || !defined(${X_}COMPILER_VERSION)
CCACHE_BUILD: Avoid ccache when looking up compiler metadata. MFC after: 2 weeks Sponsored by: Dell EMC
2018-06-27 16:58:03 +00:00
_v!= ${${cc}:N${CCACHE_BIN}} --version || echo 0.0.0
Avoid errors when MACHINE is a pseudo machine like "common".
2013-10-12 23:49:26 +00:00
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.if !defined(${X_}COMPILER_TYPE)
. if ${${cc}:T:M*gcc*}
${X_}COMPILER_TYPE:= gcc
. elif ${${cc}:T:M*clang*}
${X_}COMPILER_TYPE:= clang
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
. elif ${_v:Mgcc}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
${X_}COMPILER_TYPE:= gcc
Recognize the base/gcc compiler as GCC. The existing patterns for 'cc --version' output do not work for GCC built from the base/gcc port. Reviewed by: imp Differential Revision: https://reviews.freebsd.org/D15357
2018-05-09 15:20:39 +00:00
. elif ${_v:M\(GCC\)} || ${_v:M*GNU}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
${X_}COMPILER_TYPE:= gcc
Detect clang on macOS. The version string is slightly different.
2016-10-24 17:59:25 +00:00
. elif ${_v:Mclang} || ${_v:M(clang-*.*.*)}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
${X_}COMPILER_TYPE:= clang
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
. else
bsd.compiler.mk: Detect distribution-provided GCC when executed as cc Clang always prints "clang $VERSION" regardless of the name used to execute it, whereas GCC prints "$progname $VERSION", meaning if CC is set to cc and cc is GCC it will print "cc $VERSION". We are able to detect some of those cases since it then prints "($PKGVERSION)", where the default is "GCC", but many distributions override that to print their name and the package version number (e.g. "Debian 10.2.1-6"), so nothing tells us it's GCC other than the fact that it's not Clang (and that there's an FSF copyright disclaimer). However, GCC's -v option will always print "gcc version $VERSION", so fall back on using that to detect GCC. Whilst Clang also supports this option, we should never get here, so Clang handling is not added. Reviewed by: brooks, emaste, arichardson Differential Revision: https://reviews.freebsd.org/D28315
2021-02-09 21:40:24 +00:00
# With GCC, cc --version prints "cc $VERSION ($PKGVERSION)", so if a
# distribution overrides the default GCC PKGVERSION it is not identified.
# However, its -v output always says "gcc version" in it, so fall back on that.
_gcc_version!= ${${cc}:N${CCACHE_BIN}} -v 2>&1 | grep "gcc version"
. if !empty(_gcc_version)
${X_}COMPILER_TYPE:= gcc
. else
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.error Unable to determine compiler type for ${cc}=${${cc}}. Consider setting ${X_}COMPILER_TYPE.
bsd.compiler.mk: Detect distribution-provided GCC when executed as cc Clang always prints "clang $VERSION" regardless of the name used to execute it, whereas GCC prints "$progname $VERSION", meaning if CC is set to cc and cc is GCC it will print "cc $VERSION". We are able to detect some of those cases since it then prints "($PKGVERSION)", where the default is "GCC", but many distributions override that to print their name and the package version number (e.g. "Debian 10.2.1-6"), so nothing tells us it's GCC other than the fact that it's not Clang (and that there's an FSF copyright disclaimer). However, GCC's -v option will always print "gcc version $VERSION", so fall back on using that to detect GCC. Whilst Clang also supports this option, we should never get here, so Clang handling is not added. Reviewed by: brooks, emaste, arichardson Differential Revision: https://reviews.freebsd.org/D28315
2021-02-09 21:40:24 +00:00
. endif
.undef _gcc_version
Introduce a new make variable COMPILER_TYPE that specifies what type of compiler is being used (currently clang or gcc). COMPILER_TYPE is set in the new bsd.compiler.mk file based on the value of the CC variable or, should it prove informative, by running ${CC} --version and examining the output. To avoid negative performance impacts in the default case and correct value for COMPILER_TYPE type is determined and passed in the environment of submake instances while building world. Replace adhoc attempts at determining the compiler type by examining CC or MK_CLANG_IS_CC with checks of COMPILER_TYPE. This eliminates bootstrapping complications when first setting WITH_CLANG_IS_CC. Sponsored by: DARPA, AFRL Reviewed by: Yamaya Takashi <yamayan@kbh.biglobe.ne.jp>, imp, linimon (with some modifications post review) MFC after: 2 weeks
2012-09-13 16:00:46 +00:00
. endif
.endif
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.if !defined(${X_}COMPILER_VERSION)
Allow for compiler versions >= 10 Both clang and gcc development branches have reached version 10. Since we only parse for a single digit in the major version number, this causes COMPILER_VERSION to be set to its default of 0.0.0, meaning version checks fail with these newer compilers. Reviewed by: emaste Approved by: markj (mentor) MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D21413
2019-09-08 19:40:52 +00:00
${X_}COMPILER_VERSION!=echo "${_v:M[1-9]*.[0-9]*}" | awk -F. '{print $$1 * 10000 + $$2 * 100 + $$3;}'
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
.endif
bsd.compiler.mk: detect Apple Clang for cross-builds Apple clang uses a different versioning scheme, so if we enable or disable certain warnings for Clang 11+, those might not be supported in Apple Clang 11+. This adds 'apple-clang' to COMPILER_FEATURES, so that bootstrap tools Makefiles can avoid warnings on macOS. Reviewed By: imp Differential Revision: https://reviews.freebsd.org/D29680
2021-04-20 00:46:36 +00:00
# Detect apple clang when bootstrapping to select appropriate warning flags.
.if !defined(${X_}COMPILER_FEATURES) && ${_v:[*]:M*Apple clang version*}
${X_}COMPILER_FEATURES= apple-clang
.endif
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
.undef _v
${CC} --version doesn't need to be queried if both COMPILER_TYPE and COMPILER_VERSION are known
2014-12-06 04:02:56 +00:00
.endif
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
.if !defined(${X_}COMPILER_FREEBSD_VERSION)
CCACHE_BUILD: Avoid ccache when looking up compiler metadata. MFC after: 2 weeks Sponsored by: Dell EMC
2018-06-27 16:58:03 +00:00
${X_}COMPILER_FREEBSD_VERSION!= { echo "__FreeBSD_cc_version" | ${${cc}:N${CCACHE_BIN}} -E - 2>/dev/null || echo __FreeBSD_cc_version; } | sed -n '$$p'
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
# If we get a literal "__FreeBSD_cc_version" back then the compiler
# is a non-FreeBSD build that doesn't support it or some other error
# occurred.
.if ${${X_}COMPILER_FREEBSD_VERSION} == "__FreeBSD_cc_version"
${X_}COMPILER_FREEBSD_VERSION= unknown
.endif
.endif
Add a COMPILER_FEATURES variable that is set in bsd.compiler.mk. When the compiler is clang add c++11 to COMPILER_FEATURES. When the compiler supports c++11, default to building libc++. This will not impact the compliation of programs, but will make it easier for users with clang compiled systems to test libc++ by passing -stdlib=libc++ to their c++ compliations.
2012-09-26 17:25:15 +00:00
Fix compilation with upstream clang builtin headers. By using -nobuiltininc and adding the clang builtin headers resource dir to the end of the compiler header search path, we can still find headers such as immintrin.h but find the FreeBSD version of stddef.h/stdarg.h/.. first. This is a workaround until we are able to settle on and complete a plan to harmonize guard macros with LLVM. We've mostly worked out this on FreeBSD systems by removing select headers from the installed set of devel/llvm*, but that isn't a good solution for cross build. Submitted by: arichardson Obtained from: CheriBSD Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D17002
2020-04-06 23:38:46 +00:00
.if !defined(${X_}COMPILER_RESOURCE_DIR)
${X_}COMPILER_RESOURCE_DIR!= ${${cc}:N${CCACHE_BIN}} -print-resource-dir 2>/dev/null || echo unknown
.endif
bsd.compiler.mk: detect Apple Clang for cross-builds Apple clang uses a different versioning scheme, so if we enable or disable certain warnings for Clang 11+, those might not be supported in Apple Clang 11+. This adds 'apple-clang' to COMPILER_FEATURES, so that bootstrap tools Makefiles can avoid warnings on macOS. Reviewed By: imp Differential Revision: https://reviews.freebsd.org/D29680
2021-04-20 00:46:36 +00:00
${X_}COMPILER_FEATURES+= c++11 c++14
Remove tests for obsolete compilers in the build system Assume gcc is at least 6.4, the oldest xtoolchain in the ports tree. Assume clang is at least 6, which was in 11.2-RELEASE. Drop conditions for older compilers. Reviewed by: imp (earlier version), emaste, jhb MFC after: 2 weeks Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D24802
2020-05-12 15:22:40 +00:00
.if ${${X_}COMPILER_TYPE} == "clang" || \
Add c++14 and c++17 to COMPILER_FEATURES This will be used to gate the fusefs tests. It's a partial merge of r348281 from projects/fuse2. Reviewed by: kib, emaste MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21044
2019-07-24 15:10:09 +00:00
(${${X_}COMPILER_TYPE} == "gcc" && ${${X_}COMPILER_VERSION} >= 70000)
${X_}COMPILER_FEATURES+= c++17
.endif
bsd.compiler.mk: Add a c++20 compiler feature. This is enabled for clang versions 10+ and GCC versions 10+. Reviewed by: imp Differential Revision: https://reviews.freebsd.org/D36892
2022-11-19 04:11:59 +00:00
.if (${${X_}COMPILER_TYPE} == "clang" && ${${X_}COMPILER_VERSION} >= 100000) || \
(${${X_}COMPILER_TYPE} == "gcc" && ${${X_}COMPILER_VERSION} >= 100100)
${X_}COMPILER_FEATURES+= c++20
.endif
Enable -ftrivial-auto-var-init flags for gcc >= 12 Now that gcc >= 12 supports -ftrivial-auto-var-init, add it to bsd.compiler.mk's "init-all" feature. PR: 271047 Reviewed by: emaste MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D40208
2023-05-23 17:43:12 +00:00
.if ${${X_}COMPILER_TYPE} == "clang" || \
(${${X_}COMPILER_TYPE} == "gcc" && ${${X_}COMPILER_VERSION} >= 120000)
${X_}COMPILER_FEATURES+= init-all
.endif
Remove tests for obsolete compilers in the build system Assume gcc is at least 6.4, the oldest xtoolchain in the ports tree. Assume clang is at least 6, which was in 11.2-RELEASE. Drop conditions for older compilers. Reviewed by: imp (earlier version), emaste, jhb MFC after: 2 weeks Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D24802
2020-05-12 15:22:40 +00:00
.if ${${X_}COMPILER_TYPE} == "clang"
Enable -ftrivial-auto-var-init flags for gcc >= 12 Now that gcc >= 12 supports -ftrivial-auto-var-init, add it to bsd.compiler.mk's "init-all" feature. PR: 271047 Reviewed by: emaste MFC after: 3 days Differential Revision: https://reviews.freebsd.org/D40208
2023-05-23 17:43:12 +00:00
${X_}COMPILER_FEATURES+= retpoline
Enable compressed debug on little-endian targets Compressed debug was enabled by default in commit c910570e7573, but broke the build on big-endian targets, and so was disabled in 89ed2ecb14ce. Older versions of LLD fail with big-endian compressed debug sections. This was fixed in LLD upstream (commit c6ebc651b6fa) and merged to FreeBSD main (commit d69d07569ee2) by dim. External toolchains (e.g. the llvm12 package) will not yet have the fix. These may be used to link against base system .a archives, so compressed debug sections would cause trouble even though the base system is fixed. Enable compressed debug sections again, for little-endian targets only. As discussed on freebsd-hackers[1] I expect to undo this in the future (using compressed debug everywhere), once fixed versions of lld are widely available. Note that to be pedantically correct we should check both the compiler and the linker for compressed debug support, but given the external toolchain constraint the extra complexity does not seem worthwhile. [1] https://lists.freebsd.org/archives/freebsd-hackers/2021-August/000188.html PR: 257638 Reported by: jrtc27 [impact of .a archives] Discussed with: imp Relnotes: Yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D31454 Revert "Disable compressed debug by default" This reverts commit 89ed2ecb14ceabc27883282cf96559a9e7d52717.
2021-08-07 14:46:07 +00:00
# PR257638 lld fails with BE compressed debug. Fixed in main but external tool
# chains will initially not have the fix. For now limit the feature to LE
# targets.
bsd.compiler.mk: Fix cross-building from non-FreeBSD On non-FreeBSD, the various MACHINE variables for the host when bootstrapping can be missing or not match FreeBSD's naming, causing bsd.endian.mk to be unable to infer the endianness. Work around this by assuming it's unsupported. Note that we can't check BOOTSTRAPPING here as Makefile.inc1 includes bsd.compiler.mk before that is set, and so we are unable to catch errors during buildworld itself when cross-building and bsd.endian.mk failed, but such errors should also show up when building on FreeBSD. Fixes: 47363e99d3d3 ("Enable compressed debug on little-endian targets")
2021-08-12 22:45:09 +00:00
# When compiling bootstrap tools on non-FreeBSD, the various MACHINE variables
# for the host can be missing or not match FreeBSD's naming (e.g. Linux/amd64
# reports as MACHINE=x86_64 MACHINE_ARCH=x86_64), causing TARGET_ENDIANNESS to
# be undefined; be conservative and default to off until we turn this on by
# default everywhere.
Enable compressed debug on little-endian targets Compressed debug was enabled by default in commit c910570e7573, but broke the build on big-endian targets, and so was disabled in 89ed2ecb14ce. Older versions of LLD fail with big-endian compressed debug sections. This was fixed in LLD upstream (commit c6ebc651b6fa) and merged to FreeBSD main (commit d69d07569ee2) by dim. External toolchains (e.g. the llvm12 package) will not yet have the fix. These may be used to link against base system .a archives, so compressed debug sections would cause trouble even though the base system is fixed. Enable compressed debug sections again, for little-endian targets only. As discussed on freebsd-hackers[1] I expect to undo this in the future (using compressed debug everywhere), once fixed versions of lld are widely available. Note that to be pedantically correct we should check both the compiler and the linker for compressed debug support, but given the external toolchain constraint the extra complexity does not seem worthwhile. [1] https://lists.freebsd.org/archives/freebsd-hackers/2021-August/000188.html PR: 257638 Reported by: jrtc27 [impact of .a archives] Discussed with: imp Relnotes: Yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D31454 Revert "Disable compressed debug by default" This reverts commit 89ed2ecb14ceabc27883282cf96559a9e7d52717.
2021-08-07 14:46:07 +00:00
.include <bsd.endian.mk>
bsd.compiler.mk: Fix cross-building from non-FreeBSD On non-FreeBSD, the various MACHINE variables for the host when bootstrapping can be missing or not match FreeBSD's naming, causing bsd.endian.mk to be unable to infer the endianness. Work around this by assuming it's unsupported. Note that we can't check BOOTSTRAPPING here as Makefile.inc1 includes bsd.compiler.mk before that is set, and so we are unable to catch errors during buildworld itself when cross-building and bsd.endian.mk failed, but such errors should also show up when building on FreeBSD. Fixes: 47363e99d3d3 ("Enable compressed debug on little-endian targets")
2021-08-12 22:45:09 +00:00
.if (${.MAKE.OS} == "FreeBSD" || defined(TARGET_ENDIANNESS)) && \
${TARGET_ENDIANNESS} == "1234"
Enable compressed debug on little-endian targets Compressed debug was enabled by default in commit c910570e7573, but broke the build on big-endian targets, and so was disabled in 89ed2ecb14ce. Older versions of LLD fail with big-endian compressed debug sections. This was fixed in LLD upstream (commit c6ebc651b6fa) and merged to FreeBSD main (commit d69d07569ee2) by dim. External toolchains (e.g. the llvm12 package) will not yet have the fix. These may be used to link against base system .a archives, so compressed debug sections would cause trouble even though the base system is fixed. Enable compressed debug sections again, for little-endian targets only. As discussed on freebsd-hackers[1] I expect to undo this in the future (using compressed debug everywhere), once fixed versions of lld are widely available. Note that to be pedantically correct we should check both the compiler and the linker for compressed debug support, but given the external toolchain constraint the extra complexity does not seem worthwhile. [1] https://lists.freebsd.org/archives/freebsd-hackers/2021-August/000188.html PR: 257638 Reported by: jrtc27 [impact of .a archives] Discussed with: imp Relnotes: Yes Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D31454 Revert "Disable compressed debug by default" This reverts commit 89ed2ecb14ceabc27883282cf96559a9e7d52717.
2021-08-07 14:46:07 +00:00
${X_}COMPILER_FEATURES+= compressed-debug
.endif
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.endif
Add fileprefixmap compiler feature -ffile-prefix-map=<old>=<new> is a compiler feature first added in GCC 8, and implemented for clang 10. It remaps old paths to new paths in both debug information and __FILE__ and __BASE_FILE__ macros. It can be used to improve reproducibility or to hide local system directories. I intend to use it to replace the real source directory and real object directory with constant values across all builds. Reviewed by: brooks Differential Revision: https://reviews.freebsd.org/D28632
2021-02-17 22:26:49 +00:00
.if ${${X_}COMPILER_TYPE} == "clang" && ${${X_}COMPILER_VERSION} >= 100000 || \
(${${X_}COMPILER_TYPE} == "gcc" && ${${X_}COMPILER_VERSION} >= 80100)
${X_}COMPILER_FEATURES+= fileprefixmap
.endif
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
Add an arm64 sha512 feature check This is only available in recent toolchains. Check for it so we can use it in libmd. Reviewed by: emaste Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D33166
2021-11-29 17:21:31 +00:00
.if (${${X_}COMPILER_TYPE} == "clang" && ${${X_}COMPILER_VERSION} >= 130000) || \
(${${X_}COMPILER_TYPE} == "gcc" && ${${X_}COMPILER_VERSION} >= 90000)
# AArch64 sha512 intrinsics are supported (and have been tested) in
# clang 13 and gcc 9.
${X_}COMPILER_FEATURES+= aarch64-sha512
.endif
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.else
# Use CC's values
X_COMPILER_TYPE= ${COMPILER_TYPE}
X_COMPILER_VERSION= ${COMPILER_VERSION}
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
X_COMPILER_FREEBSD_VERSION= ${COMPILER_FREEBSD_VERSION}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
X_COMPILER_FEATURES= ${COMPILER_FEATURES}
Fix compilation with upstream clang builtin headers. By using -nobuiltininc and adding the clang builtin headers resource dir to the end of the compiler header search path, we can still find headers such as immintrin.h but find the FreeBSD version of stddef.h/stdarg.h/.. first. This is a workaround until we are able to settle on and complete a plan to harmonize guard macros with LLVM. We've mostly worked out this on FreeBSD systems by removing select headers from the installed set of devel/llvm*, but that isn't a good solution for cross build. Submitted by: arichardson Obtained from: CheriBSD Sponsored by: DARPA, AFRL Differential Revision: https://reviews.freebsd.org/D17002
2020-04-06 23:38:46 +00:00
X_COMPILER_RESOURCE_DIR= ${COMPILER_RESOURCE_DIR}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.endif # ${cc} == "CC" || (${cc} == "XCC" && ${XCC} != ${CC})
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
# Export the values so sub-makes don't have to look them up again, using the
# hash key computed above.
.for var in ${_exported_vars}
Use __ as the separator for the exported vars in bsd.compiler/linker.mk By using '__' instead of '.' as the separator we can also support systems that use dash as /bin/sh (it's the default shell on Ubuntu/Debian). Dash will unset any environment variables that use a non alphanumeric+undedscore character and therefore submakes will fail to import the COMPILER_* variables if we use '.' as the separator. Reviewed By: emaste Differential Revision: https://reviews.freebsd.org/D22381
2019-11-15 16:43:36 +00:00
${var}__${${X_}_cc_hash}:= ${${var}}
.export-env ${var}__${${X_}_cc_hash}
.undef ${var}__${${X_}_cc_hash}
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
.endfor
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.endif # ${cc} == "CC" || !empty(XCC)
.endfor # .for cc in CC XCC
bsd.compiler.mk was implicitly included by bsd.own.mk in historical versions. With its movement to src.opts.mk, bsd.prog.mk was testing COMPILER_TYPE without including the bsd.compiler.mk anymore. In the source tree, this caused no problems, for reasons that aren't clear, but does cause problems outside of the source tree. Allow bsd.compiler.mk to be included multiple times safely, and always include bsd.compiler.mk at the top of bsd.prog.mk. Resist the urge to put it in bsd.init.mk, since that would reintroduce the implicit include.
2014-05-07 18:15:02 +00:00
Add WITH_SYSTEM_LINKER, on by default, that avoids building lld when possible. This works similar to WITH_SYSTEM_COMPILER added in r300354. It only supports lld via WITH_LLD_BOOTSTRAP. When both SYSTEM_COMPILER and SYSTEM_LINKER logic passes then libclang will not build in cross-tools. If either check fails though then libclang is built. The .info is reworked to notify when libclang will be built since if either clang or lld needs to be rebuilt, but not the other, the notification can lead to confusion on why "clang is building". -fuse-ld= is not used with this method so some combinations of compiler and linker are expected to fail. A new 'make test-system-linker' target is added to see the logic results. Makefile.inc1: CROSS_BINUTILS_PREFIX support had to be moved higher up so that XLD could be set and MK_LLD_BOOTSTRAP disabled before checking SYSTEM_LINKER logic as done with SYSTEM_COMPILER. This also required moving where bsd.linker.mk was read since XLD needs to be set before parsing it. This creates a situation where src.opts.mk can not test LINKER_FEATURES or add LLD_BOOTSTAP to BROKEN_OPTIONS. Reviewed by: emaste (earlier version) Sponsored by: Dell EMC Differential Revision: https://reviews.freebsd.org/D15894
2018-06-20 16:10:14 +00:00
.if !defined(_NO_INCLUDE_LINKERMK)
Add basic bsd.linker.mk auto included from bsd.compiler.mk. This will provide LINKER_TYPE and LINKER_VERSION. MFC after: 2 weeks Reviewed by: emaste Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D11308
2017-06-22 21:03:20 +00:00
.include <bsd.linker.mk>
Add WITH_SYSTEM_LINKER, on by default, that avoids building lld when possible. This works similar to WITH_SYSTEM_COMPILER added in r300354. It only supports lld via WITH_LLD_BOOTSTRAP. When both SYSTEM_COMPILER and SYSTEM_LINKER logic passes then libclang will not build in cross-tools. If either check fails though then libclang is built. The .info is reworked to notify when libclang will be built since if either clang or lld needs to be rebuilt, but not the other, the notification can lead to confusion on why "clang is building". -fuse-ld= is not used with this method so some combinations of compiler and linker are expected to fail. A new 'make test-system-linker' target is added to see the logic results. Makefile.inc1: CROSS_BINUTILS_PREFIX support had to be moved higher up so that XLD could be set and MK_LLD_BOOTSTRAP disabled before checking SYSTEM_LINKER logic as done with SYSTEM_COMPILER. This also required moving where bsd.linker.mk was read since XLD needs to be set before parsing it. This creates a situation where src.opts.mk can not test LINKER_FEATURES or add LLD_BOOTSTAP to BROKEN_OPTIONS. Reviewed by: emaste (earlier version) Sponsored by: Dell EMC Differential Revision: https://reviews.freebsd.org/D15894
2018-06-20 16:10:14 +00:00
.endif
Don't run cc --version during cleandir/obj stages This will no work when there is no cc in $PATH (which is the case before the cross-tools stage once we no longer inherit $PATH in $WMAKE). The variables set by bsd.compiler.mk/bsd.linker.mk are not needed in these stages so this avoids a little bit of makefile parsing. Reviewed By: emaste Approved By: jhb (mentor) Differential Revision: https://reviews.freebsd.org/D16814
2018-10-31 10:45:28 +00:00
.endif # defined(_NO_INCLUDE_COMPILERMK)
bsd.compiler.mk was implicitly included by bsd.own.mk in historical versions. With its movement to src.opts.mk, bsd.prog.mk was testing COMPILER_TYPE without including the bsd.compiler.mk anymore. In the source tree, this caused no problems, for reasons that aren't clear, but does cause problems outside of the source tree. Allow bsd.compiler.mk to be included multiple times safely, and always include bsd.compiler.mk at the top of bsd.prog.mk. Resist the urge to put it in bsd.init.mk, since that would reintroduce the implicit include.
2014-05-07 18:15:02 +00:00
.endif # !target(__<bsd.compiler.mk>__)
Reference in New Issue Copy Permalink