d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
bb66a3e948
freebsd-skq/share/mk/bsd.compiler.mk

245 lines
8.9 KiB
Makefile
Raw Normal View History

Introduce a new make variable COMPILER_TYPE that specifies what type of compiler is being used (currently clang or gcc). COMPILER_TYPE is set in the new bsd.compiler.mk file based on the value of the CC variable or, should it prove informative, by running ${CC} --version and examining the output. To avoid negative performance impacts in the default case and correct value for COMPILER_TYPE type is determined and passed in the environment of submake instances while building world. Replace adhoc attempts at determining the compiler type by examining CC or MK_CLANG_IS_CC with checks of COMPILER_TYPE. This eliminates bootstrapping complications when first setting WITH_CLANG_IS_CC. Sponsored by: DARPA, AFRL Reviewed by: Yamaya Takashi <yamayan@kbh.biglobe.ne.jp>, imp, linimon (with some modifications post review) MFC after: 2 weeks
2012-09-13 16:00:46 +00:00
# $FreeBSD$
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
# Setup variables for the compiler
#
Fix typos in comments and wrap to <80 columns MFC after: 3 days
2014-12-06 03:12:57 +00:00
# COMPILER_TYPE is the major type of compiler. Currently gcc and clang support
# automatic detection. Other compiler types can be shoe-horned in, but require
# explicit setting of the compiler type. The compiler type can also be set
# explicitly if, say, you install gcc as clang...
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
#
Fix typos in comments and wrap to <80 columns MFC after: 3 days
2014-12-06 03:12:57 +00:00
# COMPILER_VERSION is a numeric constant equal to:
# major * 10000 + minor * 100 + tiny
Fix spelling error. Reported by: arc Sponsored by: Dell EMC Isilon
2017-06-22 20:32:23 +00:00
# It too can be overridden on the command line. When testing it, be sure to
Fix typos in comments and wrap to <80 columns MFC after: 3 days
2014-12-06 03:12:57 +00:00
# make sure that you are limiting the test to a specific compiler. Testing
# against 30300 for gcc likely isn't what you wanted (since versions of gcc
# prior to 4.2 likely have no prayer of working).
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
#
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
# COMPILER_FREEBSD_VERSION is the compiler's __FreeBSD_cc_version value.
#
Fix typos in comments and wrap to <80 columns MFC after: 3 days
2014-12-06 03:12:57 +00:00
# COMPILER_FEATURES will contain one or more of the following, based on
# compiler support for that feature:
#
Add c++14 and c++17 to COMPILER_FEATURES This will be used to gate the fusefs tests. It's a partial merge of r348281 from projects/fuse2. Reviewed by: kib, emaste MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21044
2019-07-24 15:10:09 +00:00
# - c++17: supports full (or nearly full) C++17 programming environment.
# - c++14: supports full (or nearly full) C++14 programming environment.
add retpoline compiler and linker feature flags These features indicate that the compiler and linker support the retpoline speculative execution vulnerability (CVE-2017-5715) mitigation. Reviewed by: dim, imp Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D14228
2018-02-07 14:50:06 +00:00
# - c++11: supports full (or nearly full) C++11 programming environment.
# - retpoline: supports the retpoline speculative execution vulnerability
# mitigation.
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
#
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
# These variables with an X_ prefix will also be provided if XCC is set.
#
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
# This file may be included multiple times, but only has effect the first time.
#
bsd.compiler.mk was implicitly included by bsd.own.mk in historical versions. With its movement to src.opts.mk, bsd.prog.mk was testing COMPILER_TYPE without including the bsd.compiler.mk anymore. In the source tree, this caused no problems, for reasons that aren't clear, but does cause problems outside of the source tree. Allow bsd.compiler.mk to be included multiple times safely, and always include bsd.compiler.mk at the top of bsd.prog.mk. Resist the urge to put it in bsd.init.mk, since that would reintroduce the implicit include.
2014-05-07 18:15:02 +00:00
.if !target(__<bsd.compiler.mk>__)
__<bsd.compiler.mk>__:
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
.include <bsd.opts.mk>
Don't run cc --version during cleandir/obj stages This will no work when there is no cc in $PATH (which is the case before the cross-tools stage once we no longer inherit $PATH in $WMAKE). The variables set by bsd.compiler.mk/bsd.linker.mk are not needed in these stages so this avoids a little bit of makefile parsing. Reviewed By: emaste Approved By: jhb (mentor) Differential Revision: https://reviews.freebsd.org/D16814
2018-10-31 10:45:28 +00:00
.if defined(_NO_INCLUDE_COMPILERMK)
# If _NO_INCLUDE_COMPILERMK is set we are doing a make obj/cleandir/cleanobj
# and might not have a valid compiler in $PATH yet. In this case just set the
# variables that are expected by the other .mk files and return
COMPILER_TYPE=none
X_COMPILER_TYPE=none
COMPILER_VERSION=0
X_COMPILER_VERSION=0
COMPILER_FEATURES=none
.else
CCACHE_BUILD: Allow setting CCACHE_BUILD_TYPE=wrapper. This uses the /usr/local/libexec/ccache/<cc,c++> wrappers rather than modifying CC to be '/usr/local/bin/ccache cc'. Some forms of compilation do not support the 'command' type. Sponsored by: Dell EMC Isilon
2017-08-01 16:15:08 +00:00
# command = /usr/local/bin/ccache cc ...
# wrapper = /usr/local/libexec/ccache/cc ...
CCACHE_BUILD_TYPE?= command
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
# Handle ccache after CC is determined, but not if CC/CXX are already
# overridden with a manual setup.
.if ${MK_CCACHE_BUILD:Uno} == "yes" && \
Reduce exec and fstat overhead for non-build targets. This is mostly targetting 'make showconfig' and 'make test-system-*' for the benefit of makeman and universe-one-clang work. Sponsored by: Dell EMC
2018-06-20 17:20:39 +00:00
!make(test-system-*) && !make(print-dir) && !make(showconfig) && \
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
(${CC:M*ccache/world/*} == "" || ${CXX:M*ccache/world/*} == "")
# CC is always prepended with the ccache wrapper rather than modifying
# PATH since it is more clear that ccache is used and avoids wasting time
# for mkdep/linking/asm builds.
LOCALBASE?= /usr/local
CCACHE_WRAPPER_PATH?= ${LOCALBASE}/libexec/ccache
CCACHE_BIN?= ${LOCALBASE}/bin/ccache
.if exists(${CCACHE_BIN})
# Export to ensure sub-makes can filter it out for mkdep/linking and
# to chain down into kernel build which won't include this file.
.export CCACHE_BIN
# Expand and export some variables so they may be based on make vars.
# This allows doing something like the following in the environment:
# CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
.for var in CCACHE_LOGFILE CCACHE_BASEDIR
.if defined(${var})
${var}:= ${${var}}
.export ${var}
.endif
.endfor
# Handle bootstrapped compiler changes properly by hashing their content
# rather than checking mtime. For external compilers it should be safe
# to use the more optimal mtime check.
# XXX: CCACHE_COMPILERCHECK= string:<compiler_version, compiler_build_rev, compiler_patch_rev, compiler_default_target, compiler_default_sysroot>
.if ${CC:N${CCACHE_BIN}:[1]:M/*} == ""
CCACHE_COMPILERCHECK?= content
.else
CCACHE_COMPILERCHECK?= mtime
.endif
.export CCACHE_COMPILERCHECK
# Ensure no bogus CCACHE_PATH leaks in which might avoid the in-tree compiler.
CCACHE_BUILD: Only export CCACHE_PATH= if it was already set with a value. Older ccache don't work with an empty CCACHE_PATH value. They will error with: ccache: FATAL: Could not find compiler "cc" in PATH make: "/mnt/bdrewery/git/onefs/src/share/mk/bsd.compiler.mk" line 134: Unable to determine compiler type for /usr/local/bin/ccache cc. Consider setting COMPILER_TYPE. Sponsored by: EMC / Isilon Storage Division
2015-12-08 20:20:40 +00:00
.if !empty(CCACHE_PATH)
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
CCACHE_PATH=
.export CCACHE_PATH
CCACHE_BUILD: Only export CCACHE_PATH= if it was already set with a value. Older ccache don't work with an empty CCACHE_PATH value. They will error with: ccache: FATAL: Could not find compiler "cc" in PATH make: "/mnt/bdrewery/git/onefs/src/share/mk/bsd.compiler.mk" line 134: Unable to determine compiler type for /usr/local/bin/ccache cc. Consider setting COMPILER_TYPE. Sponsored by: EMC / Isilon Storage Division
2015-12-08 20:20:40 +00:00
.endif
CCACHE_BUILD: Allow setting CCACHE_BUILD_TYPE=wrapper. This uses the /usr/local/libexec/ccache/<cc,c++> wrappers rather than modifying CC to be '/usr/local/bin/ccache cc'. Some forms of compilation do not support the 'command' type. Sponsored by: Dell EMC Isilon
2017-08-01 16:15:08 +00:00
.if ${CCACHE_BUILD_TYPE} == "command"
# Remove ccache from the PATH to prevent double calls and wasted CPP/LD time.
PATH:= ${PATH:C,:?${CCACHE_WRAPPER_PATH}(/world)?(:$)?,,g}
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
# Override various toolchain vars.
.for var in CC CXX HOST_CC HOST_CXX
.if defined(${var}) && ${${var}:M${CCACHE_BIN}} == ""
${var}:= ${CCACHE_BIN} ${${var}}
.endif
.endfor
CCACHE_BUILD: Follow-up r321880: Fix some PATH issues with buildworld. - bsd.compiler.mk: Must ensure that the CCACHE_WRAPPER_PATH comes first in PATH. - Makefile.inc1: Must prepend the CCACHE_WRAPPER_PATH into BPATH as it overrides the PATH set in bsd.compiler.mk in sub-makes. The PATH set in bsd.compiler.mk is not exported and doing so would cause it to then override the BPATH set from environment. The only sane solution is to prepend into BPATH as needed. CCACHE_PATH could possibly be used for some of this as well. Sponsored by: Dell EMC Isilon
2017-08-01 18:26:20 +00:00
.else
# Need to ensure CCACHE_WRAPPER_PATH is the first in ${PATH}
PATH:= ${PATH:C,:?${CCACHE_WRAPPER_PATH}(/world)?(:$)?,,g}
CCACHE_BUILD: Allow setting CCACHE_BUILD_TYPE=wrapper. This uses the /usr/local/libexec/ccache/<cc,c++> wrappers rather than modifying CC to be '/usr/local/bin/ccache cc'. Some forms of compilation do not support the 'command' type. Sponsored by: Dell EMC Isilon
2017-08-01 16:15:08 +00:00
PATH:= ${CCACHE_WRAPPER_PATH}:${PATH}
CCACHE_BUILD: Follow-up r321880: Fix some PATH issues with buildworld. - bsd.compiler.mk: Must ensure that the CCACHE_WRAPPER_PATH comes first in PATH. - Makefile.inc1: Must prepend the CCACHE_WRAPPER_PATH into BPATH as it overrides the PATH set in bsd.compiler.mk in sub-makes. The PATH set in bsd.compiler.mk is not exported and doing so would cause it to then override the BPATH set from environment. The only sane solution is to prepend into BPATH as needed. CCACHE_PATH could possibly be used for some of this as well. Sponsored by: Dell EMC Isilon
2017-08-01 18:26:20 +00:00
CCACHE_WRAPPER_PATH_PFX= ${CCACHE_WRAPPER_PATH}:
CCACHE_BUILD: Allow setting CCACHE_BUILD_TYPE=wrapper. This uses the /usr/local/libexec/ccache/<cc,c++> wrappers rather than modifying CC to be '/usr/local/bin/ccache cc'. Some forms of compilation do not support the 'command' type. Sponsored by: Dell EMC Isilon
2017-08-01 16:15:08 +00:00
.endif # ${CCACHE_BUILD_TYPE} == "command"
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
# GCC does not need the CCACHE_CPP2 hack enabled by default in devel/ccache.
# The port enables it due to ccache passing preprocessed C to clang
# which fails with -Wparentheses-equality, -Wtautological-compare, and
# -Wself-assign on macro-expanded lines.
.if defined(COMPILER_TYPE) && ${COMPILER_TYPE} == "gcc"
CCACHE_NOCPP2= 1
.export CCACHE_NOCPP2
.endif
# Canonicalize CCACHE_DIR for meta mode usage.
CCACHE_BUILD+META_MODE: Lookup and canonicalize the default CCACHE_DIR. This is done to prevent not having CCACHE_DIR causing meta mode with filemon to see stat changes in the ccache dir and cause rebuilds. Sponsored by: EMC / Isilon Storage Division
2016-03-01 22:25:54 +00:00
.if !defined(CCACHE_DIR)
CCACHE_DIR!= ${CCACHE_BIN} -p | awk '$$2 == "cache_dir" {print $$4}'
.export CCACHE_DIR
.endif
.if !empty(CCACHE_DIR) && empty(.MAKE.META.IGNORE_PATHS:M${CCACHE_DIR})
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
CCACHE_DIR:= ${CCACHE_DIR:tA}
.MAKE.META.IGNORE_PATHS+= ${CCACHE_DIR}
.export CCACHE_DIR
.endif
WITH_CCACHE_BUILD + WITH_META_MODE: Ignore ccache changes. Ccache will not affect the output of the objects, so just ignore it for meta mode handling. This avoids having everything rebuild if ccache is updated. Sponsored by: EMC / Isilon Storage Division
2016-06-03 19:25:45 +00:00
# ccache doesn't affect build output so let it slide for meta mode
# comparisons.
.MAKE.META.IGNORE_PATHS+= ${CCACHE_BIN}
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
ccache-print-options: .PHONY
@${CCACHE_BIN} -p
.endif # exists(${CCACHE_BIN})
.endif # ${MK_CCACHE_BUILD} == "yes"
Only compute the X_COMPILER_*/X_LINKER_* variables when needed When building CheriBSD we have to set XLD/XCC/XCFLAGS on the command line. This triggers the $XCC != $CC case in bsd.compiler.mk (and the same for LD in bsd.linker.mk) which causes it to call ${XCC} --version and ${XLD} --version (plus various awk+sed+echo calls) in every subdirectory. For incremental builds and stages that only walk the source tree this is often the majority of the time spent in that directory. By only computing the value of the X_COMPILER_*/X_LINKER_* variables if _WANT_TOOLCHAIN_CROSS_VARS is set we can reduce the number of cc/ld calls to once per build stage instead of once per recursive make. With this change (and no changes to the sources) the `make includes` stage now takes 28 seconds at -j1 instead of 86 seconds. Approved By: brooks (mentor) Differential Revision: https://reviews.freebsd.org/D17046
2018-10-23 06:31:25 +00:00
_cc_vars=CC $${_empty_var_}
.if !empty(_WANT_TOOLCHAIN_CROSS_VARS)
# Only the toplevel makefile needs to compute the X_COMPILER_* variables.
# Skipping the computation of the unused X_COMPILER_* in the subdirectory
# makefiles can save a noticeable amount of time when walking the whole source
# tree (e.g. during make includes, etc.).
_cc_vars+=XCC X_
.endif
.for cc X_ in ${_cc_vars}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.if ${cc} == "CC" || !empty(XCC)
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
# Try to import COMPILER_TYPE and COMPILER_VERSION from parent make.
# The value is only used/exported for the same environment that impacts
# CC and COMPILER_* settings here.
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
_exported_vars= ${X_}COMPILER_TYPE ${X_}COMPILER_VERSION \
${X_}COMPILER_FREEBSD_VERSION
GCC External: Revert r300886, r300904, r300917, r300918 The fix in r300873 is mostly enough. A fix for lib32 will be committed.separately.
2016-05-29 06:20:15 +00:00
${X_}_cc_hash= ${${cc}}${MACHINE}${PATH}
${X_}_cc_hash:= ${${X_}_cc_hash:hash}
Improve safety of caching from r289659 by only importing of none of the variables are already set. This should cover odd cases such as the COMPILER_TYPE override in lib/csu/powerpc64. X-MFC-With: r289659 MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division
2015-10-20 20:37:00 +00:00
# Only import if none of the vars are set somehow else.
_can_export= yes
.for var in ${_exported_vars}
.if defined(${var})
_can_export= no
.endif
.endfor
.if ${_can_export} == yes
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
.for var in ${_exported_vars}
GCC External: Revert r300886, r300904, r300917, r300918 The fix in r300873 is mostly enough. A fix for lib32 will be committed.separately.
2016-05-29 06:20:15 +00:00
.if defined(${var}.${${X_}_cc_hash})
${var}= ${${var}.${${X_}_cc_hash}}
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
.endif
.endfor
Improve safety of caching from r289659 by only importing of none of the variables are already set. This should cover odd cases such as the COMPILER_TYPE override in lib/csu/powerpc64. X-MFC-With: r289659 MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division
2015-10-20 20:37:00 +00:00
.endif
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.if ${cc} == "CC" || (${cc} == "XCC" && ${XCC} != ${CC})
Avoid errors when MACHINE is a pseudo machine like "common".
2013-10-12 23:49:26 +00:00
.if ${MACHINE} == "common"
Mention what 'common' is for
2015-06-10 19:36:53 +00:00
# common is a pseudo machine for architecture independent
# generated files - thus there is no compiler.
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
${X_}COMPILER_TYPE= none
${X_}COMPILER_VERSION= 0
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
${X_}COMPILER_FREEBSD_VERSION= 0
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.elif !defined(${X_}COMPILER_TYPE) || !defined(${X_}COMPILER_VERSION)
CCACHE_BUILD: Avoid ccache when looking up compiler metadata. MFC after: 2 weeks Sponsored by: Dell EMC
2018-06-27 16:58:03 +00:00
_v!= ${${cc}:N${CCACHE_BIN}} --version || echo 0.0.0
Avoid errors when MACHINE is a pseudo machine like "common".
2013-10-12 23:49:26 +00:00
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.if !defined(${X_}COMPILER_TYPE)
. if ${${cc}:T:M*gcc*}
${X_}COMPILER_TYPE:= gcc
. elif ${${cc}:T:M*clang*}
${X_}COMPILER_TYPE:= clang
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
. elif ${_v:Mgcc}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
${X_}COMPILER_TYPE:= gcc
Recognize the base/gcc compiler as GCC. The existing patterns for 'cc --version' output do not work for GCC built from the base/gcc port. Reviewed by: imp Differential Revision: https://reviews.freebsd.org/D15357
2018-05-09 15:20:39 +00:00
. elif ${_v:M\(GCC\)} || ${_v:M*GNU}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
${X_}COMPILER_TYPE:= gcc
Detect clang on macOS. The version string is slightly different.
2016-10-24 17:59:25 +00:00
. elif ${_v:Mclang} || ${_v:M(clang-*.*.*)}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
${X_}COMPILER_TYPE:= clang
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
. else
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.error Unable to determine compiler type for ${cc}=${${cc}}. Consider setting ${X_}COMPILER_TYPE.
Introduce a new make variable COMPILER_TYPE that specifies what type of compiler is being used (currently clang or gcc). COMPILER_TYPE is set in the new bsd.compiler.mk file based on the value of the CC variable or, should it prove informative, by running ${CC} --version and examining the output. To avoid negative performance impacts in the default case and correct value for COMPILER_TYPE type is determined and passed in the environment of submake instances while building world. Replace adhoc attempts at determining the compiler type by examining CC or MK_CLANG_IS_CC with checks of COMPILER_TYPE. This eliminates bootstrapping complications when first setting WITH_CLANG_IS_CC. Sponsored by: DARPA, AFRL Reviewed by: Yamaya Takashi <yamayan@kbh.biglobe.ne.jp>, imp, linimon (with some modifications post review) MFC after: 2 weeks
2012-09-13 16:00:46 +00:00
. endif
.endif
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.if !defined(${X_}COMPILER_VERSION)
${X_}COMPILER_VERSION!=echo "${_v:M[1-9].[0-9]*}" | awk -F. '{print $$1 * 10000 + $$2 * 100 + $$3;}'
Support, to the extent we generate proper command lines, compiling with clang 3.3. Useful for test building -current on a -stable system in individual directories. Potentially useful if we ever want to support, say, gcc 4.8 or 4.9's new warnings when building with an external toolchain (but such support not yet committed). Document the bsd.compiler.mk interface.
2014-05-10 16:37:44 +00:00
.endif
.undef _v
${CC} --version doesn't need to be queried if both COMPILER_TYPE and COMPILER_VERSION are known
2014-12-06 04:02:56 +00:00
.endif
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
.if !defined(${X_}COMPILER_FREEBSD_VERSION)
CCACHE_BUILD: Avoid ccache when looking up compiler metadata. MFC after: 2 weeks Sponsored by: Dell EMC
2018-06-27 16:58:03 +00:00
${X_}COMPILER_FREEBSD_VERSION!= { echo "__FreeBSD_cc_version" | ${${cc}:N${CCACHE_BIN}} -E - 2>/dev/null || echo __FreeBSD_cc_version; } | sed -n '$$p'
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
# If we get a literal "__FreeBSD_cc_version" back then the compiler
# is a non-FreeBSD build that doesn't support it or some other error
# occurred.
.if ${${X_}COMPILER_FREEBSD_VERSION} == "__FreeBSD_cc_version"
${X_}COMPILER_FREEBSD_VERSION= unknown
.endif
.endif
Add a COMPILER_FEATURES variable that is set in bsd.compiler.mk. When the compiler is clang add c++11 to COMPILER_FEATURES. When the compiler supports c++11, default to building libc++. This will not impact the compliation of programs, but will make it easier for users with clang compiled systems to test libc++ by passing -stdlib=libc++ to their c++ compliations.
2012-09-26 17:25:15 +00:00
add retpoline compiler and linker feature flags These features indicate that the compiler and linker support the retpoline speculative execution vulnerability (CVE-2017-5715) mitigation. Reviewed by: dim, imp Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D14228
2018-02-07 14:50:06 +00:00
${X_}COMPILER_FEATURES=
Add c++14 and c++17 to COMPILER_FEATURES This will be used to gate the fusefs tests. It's a partial merge of r348281 from projects/fuse2. Reviewed by: kib, emaste MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21044
2019-07-24 15:10:09 +00:00
.if (${${X_}COMPILER_TYPE} == "clang" && ${${X_}COMPILER_VERSION} >= 30300) || \
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
(${${X_}COMPILER_TYPE} == "gcc" && ${${X_}COMPILER_VERSION} >= 40800)
add retpoline compiler and linker feature flags These features indicate that the compiler and linker support the retpoline speculative execution vulnerability (CVE-2017-5715) mitigation. Reviewed by: dim, imp Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D14228
2018-02-07 14:50:06 +00:00
${X_}COMPILER_FEATURES+= c++11
.endif
Add c++14 and c++17 to COMPILER_FEATURES This will be used to gate the fusefs tests. It's a partial merge of r348281 from projects/fuse2. Reviewed by: kib, emaste MFC after: 2 weeks Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21044
2019-07-24 15:10:09 +00:00
.if (${${X_}COMPILER_TYPE} == "clang" && ${${X_}COMPILER_VERSION} >= 30400) || \
(${${X_}COMPILER_TYPE} == "gcc" && ${${X_}COMPILER_VERSION} >= 50000)
${X_}COMPILER_FEATURES+= c++14
.endif
.if (${${X_}COMPILER_TYPE} == "clang" && ${${X_}COMPILER_VERSION} >= 50000) || \
(${${X_}COMPILER_TYPE} == "gcc" && ${${X_}COMPILER_VERSION} >= 70000)
${X_}COMPILER_FEATURES+= c++17
.endif
add retpoline compiler and linker feature flags These features indicate that the compiler and linker support the retpoline speculative execution vulnerability (CVE-2017-5715) mitigation. Reviewed by: dim, imp Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D14228
2018-02-07 14:50:06 +00:00
.if ${${X_}COMPILER_TYPE} == "clang" && ${${X_}COMPILER_VERSION} >= 60000
${X_}COMPILER_FEATURES+= retpoline
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.endif
.else
# Use CC's values
X_COMPILER_TYPE= ${COMPILER_TYPE}
X_COMPILER_VERSION= ${COMPILER_VERSION}
Fetch the __FreeBSD_cc_version as COMPILER_FREEBSD_VERSION. Reviewed by: brooks, bapt, dim Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6356
2016-05-21 01:32:13 +00:00
X_COMPILER_FREEBSD_VERSION= ${COMPILER_FREEBSD_VERSION}
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
X_COMPILER_FEATURES= ${COMPILER_FEATURES}
.endif # ${cc} == "CC" || (${cc} == "XCC" && ${XCC} != ${CC})
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
# Export the values so sub-makes don't have to look them up again, using the
# hash key computed above.
.for var in ${_exported_vars}
GCC External: Revert r300886, r300904, r300917, r300918 The fix in r300873 is mostly enough. A fix for lib32 will be committed.separately.
2016-05-29 06:20:15 +00:00
${var}.${${X_}_cc_hash}:= ${${var}}
.export-env ${var}.${${X_}_cc_hash}
.undef ${var}.${${X_}_cc_hash}
Pass COMPILER_TYPE and COMPILER_VERSION to sub-makes to avoid redundant lookups. This uses a special variable name based on a hash of ${CC}, ${PATH}, and ${MACHINE} to ensure that a cached value is not used if any of these values changes to use a new compiler. Before this there were 34,620 fork/exec from bsd.compiler.mk during a buildworld. After this there are 608. More improvement is needed to cache a value from the top-level before descending into subdirs in the various build phases. Reviewed by: brooks (earlier version) MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D3898
2015-10-20 20:15:25 +00:00
.endfor
Auto determine X_COMPILER_TYPE/X_COMPILER_VERSION if XCC is set. Reviewed by: brooks, bapt Sponsored by: EMC / Isilon Storage Division Differential Revision: https://reviews.freebsd.org/D6355
2016-05-21 01:32:10 +00:00
.endif # ${cc} == "CC" || !empty(XCC)
.endfor # .for cc in CC XCC
bsd.compiler.mk was implicitly included by bsd.own.mk in historical versions. With its movement to src.opts.mk, bsd.prog.mk was testing COMPILER_TYPE without including the bsd.compiler.mk anymore. In the source tree, this caused no problems, for reasons that aren't clear, but does cause problems outside of the source tree. Allow bsd.compiler.mk to be included multiple times safely, and always include bsd.compiler.mk at the top of bsd.prog.mk. Resist the urge to put it in bsd.init.mk, since that would reintroduce the implicit include.
2014-05-07 18:15:02 +00:00
Add WITH_SYSTEM_LINKER, on by default, that avoids building lld when possible. This works similar to WITH_SYSTEM_COMPILER added in r300354. It only supports lld via WITH_LLD_BOOTSTRAP. When both SYSTEM_COMPILER and SYSTEM_LINKER logic passes then libclang will not build in cross-tools. If either check fails though then libclang is built. The .info is reworked to notify when libclang will be built since if either clang or lld needs to be rebuilt, but not the other, the notification can lead to confusion on why "clang is building". -fuse-ld= is not used with this method so some combinations of compiler and linker are expected to fail. A new 'make test-system-linker' target is added to see the logic results. Makefile.inc1: CROSS_BINUTILS_PREFIX support had to be moved higher up so that XLD could be set and MK_LLD_BOOTSTRAP disabled before checking SYSTEM_LINKER logic as done with SYSTEM_COMPILER. This also required moving where bsd.linker.mk was read since XLD needs to be set before parsing it. This creates a situation where src.opts.mk can not test LINKER_FEATURES or add LLD_BOOTSTAP to BROKEN_OPTIONS. Reviewed by: emaste (earlier version) Sponsored by: Dell EMC Differential Revision: https://reviews.freebsd.org/D15894
2018-06-20 16:10:14 +00:00
.if !defined(_NO_INCLUDE_LINKERMK)
Add basic bsd.linker.mk auto included from bsd.compiler.mk. This will provide LINKER_TYPE and LINKER_VERSION. MFC after: 2 weeks Reviewed by: emaste Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D11308
2017-06-22 21:03:20 +00:00
.include <bsd.linker.mk>
Add WITH_SYSTEM_LINKER, on by default, that avoids building lld when possible. This works similar to WITH_SYSTEM_COMPILER added in r300354. It only supports lld via WITH_LLD_BOOTSTRAP. When both SYSTEM_COMPILER and SYSTEM_LINKER logic passes then libclang will not build in cross-tools. If either check fails though then libclang is built. The .info is reworked to notify when libclang will be built since if either clang or lld needs to be rebuilt, but not the other, the notification can lead to confusion on why "clang is building". -fuse-ld= is not used with this method so some combinations of compiler and linker are expected to fail. A new 'make test-system-linker' target is added to see the logic results. Makefile.inc1: CROSS_BINUTILS_PREFIX support had to be moved higher up so that XLD could be set and MK_LLD_BOOTSTRAP disabled before checking SYSTEM_LINKER logic as done with SYSTEM_COMPILER. This also required moving where bsd.linker.mk was read since XLD needs to be set before parsing it. This creates a situation where src.opts.mk can not test LINKER_FEATURES or add LLD_BOOTSTAP to BROKEN_OPTIONS. Reviewed by: emaste (earlier version) Sponsored by: Dell EMC Differential Revision: https://reviews.freebsd.org/D15894
2018-06-20 16:10:14 +00:00
.endif
Don't run cc --version during cleandir/obj stages This will no work when there is no cc in $PATH (which is the case before the cross-tools stage once we no longer inherit $PATH in $WMAKE). The variables set by bsd.compiler.mk/bsd.linker.mk are not needed in these stages so this avoids a little bit of makefile parsing. Reviewed By: emaste Approved By: jhb (mentor) Differential Revision: https://reviews.freebsd.org/D16814
2018-10-31 10:45:28 +00:00
.endif # defined(_NO_INCLUDE_COMPILERMK)
bsd.compiler.mk was implicitly included by bsd.own.mk in historical versions. With its movement to src.opts.mk, bsd.prog.mk was testing COMPILER_TYPE without including the bsd.compiler.mk anymore. In the source tree, this caused no problems, for reasons that aren't clear, but does cause problems outside of the source tree. Allow bsd.compiler.mk to be included multiple times safely, and always include bsd.compiler.mk at the top of bsd.prog.mk. Resist the urge to put it in bsd.init.mk, since that would reintroduce the implicit include.
2014-05-07 18:15:02 +00:00
.endif # !target(__<bsd.compiler.mk>__)
Reference in New Issue Copy Permalink