freebsd-nq/share/mk/bsd.prog.mk

312 lines
7.1 KiB
Makefile
Raw Normal View History

# from: @(#)bsd.prog.mk 5.26 (Berkeley) 6/25/91
1999-08-28 00:22:10 +00:00
# $FreeBSD$
1994-05-30 19:09:18 +00:00
.include <bsd.init.mk>
.include <bsd.compiler.mk>
1994-05-30 19:09:18 +00:00
.SUFFIXES: .out .o .bc .c .cc .cpp .cxx .C .m .y .l .ll .ln .s .S .asm
1994-05-30 19:09:18 +00:00
# XXX The use of COPTS in modern makefiles is discouraged.
.if defined(COPTS)
.warning ${.CURDIR}: COPTS should be CFLAGS.
CFLAGS+=${COPTS}
.endif
.if ${MK_ASSERT_DEBUG} == "no"
CFLAGS+= -DNDEBUG
NO_WERROR=
.endif
.if defined(DEBUG_FLAGS)
CFLAGS+=${DEBUG_FLAGS}
CXXFLAGS+=${DEBUG_FLAGS}
Add support for the Compact C Type (CTF) conversions throughout FreeBSD's system makefiles. Note that the CTF conversion defaults to off. We may choose to change this default later if DTrace proves popular and people are prepared to wear the compilation performance impact of compiling with debug symbols all the time. Setting NO_CTF in the make args or user environment turns off CTF conversion. Even if we choose to default CTF generation to on later, we still need NO_CTF so that the buildworld process can bootstrap the tools without needlessly generating CTF data for temporary tools. Setting WITH_CTF in the make args or user environment (and _NOT_ in /etc/make.conf) is the only way to enable CTF data conversion. Nore that this can't be implemented the same way that the WITH_ and WITHOUT_ stuff is implemented throughout the buildworld because the CTF conversion needs to work when building a simple object without a Makefile, using the default rules in sys.mk. Typing 'make test.o' with no makefile and just a source file test.c should work. Also, typing 'make WITH_CTF=1 test.o without a makefile and just a source file test.c should work and produce an object with a CTF elf section. Typing 'make WITH_CTF=1 CFLAGS=-g test.o' without a makefile and just a source file test.c should produce an object with both a CTF elf section and the debug elf sections. In the FreeBSD build where more .mk files are used than just sys.mk which is included my make by default, the use of DEBUG_FLAGS is the correct way to enable a debug build. The important thing to note here is that it is the DEBUG_FLAGS setting that prevents libraries and programs from being stripped on installation. So, for the addition of CTF data conversion, setting DEBUG_FLAGS to contain -g, without NO_CTF, will cause the ctfconvert and ctfmerge build programs to be executed also with the -g arg so that debug symbols are retained rather than being removed after the CTF data elf section has been added. Add DTrace libraries to the list of libnames.
2008-05-22 01:14:43 +00:00
.if ${MK_CTF} != "no" && ${DEBUG_FLAGS:M-g} != ""
Add support for the Compact C Type (CTF) conversions throughout FreeBSD's system makefiles. Note that the CTF conversion defaults to off. We may choose to change this default later if DTrace proves popular and people are prepared to wear the compilation performance impact of compiling with debug symbols all the time. Setting NO_CTF in the make args or user environment turns off CTF conversion. Even if we choose to default CTF generation to on later, we still need NO_CTF so that the buildworld process can bootstrap the tools without needlessly generating CTF data for temporary tools. Setting WITH_CTF in the make args or user environment (and _NOT_ in /etc/make.conf) is the only way to enable CTF data conversion. Nore that this can't be implemented the same way that the WITH_ and WITHOUT_ stuff is implemented throughout the buildworld because the CTF conversion needs to work when building a simple object without a Makefile, using the default rules in sys.mk. Typing 'make test.o' with no makefile and just a source file test.c should work. Also, typing 'make WITH_CTF=1 test.o without a makefile and just a source file test.c should work and produce an object with a CTF elf section. Typing 'make WITH_CTF=1 CFLAGS=-g test.o' without a makefile and just a source file test.c should produce an object with both a CTF elf section and the debug elf sections. In the FreeBSD build where more .mk files are used than just sys.mk which is included my make by default, the use of DEBUG_FLAGS is the correct way to enable a debug build. The important thing to note here is that it is the DEBUG_FLAGS setting that prevents libraries and programs from being stripped on installation. So, for the addition of CTF data conversion, setting DEBUG_FLAGS to contain -g, without NO_CTF, will cause the ctfconvert and ctfmerge build programs to be executed also with the -g arg so that debug symbols are retained rather than being removed after the CTF data elf section has been added. Add DTrace libraries to the list of libnames.
2008-05-22 01:14:43 +00:00
CTFFLAGS+= -g
.endif
.endif
.if defined(PROG_CXX)
PROG= ${PROG_CXX}
.endif
.if !empty(LDFLAGS:M-Wl,*--oformat,*) || !empty(LDFLAGS:M-static)
MK_DEBUG_FILES= no
.endif
.if defined(CRUNCH_CFLAGS)
CFLAGS+=${CRUNCH_CFLAGS}
.else
.if ${MK_DEBUG_FILES} != "no" && empty(DEBUG_FLAGS:M-g) && \
empty(DEBUG_FLAGS:M-gdwarf-*)
CFLAGS+= -g
CTFFLAGS+= -g
.endif
.endif
1994-05-30 19:09:18 +00:00
.if !defined(DEBUG_FLAGS)
1994-05-30 19:09:18 +00:00
STRIP?= -s
.endif
1994-05-30 19:09:18 +00:00
.if defined(NO_ROOT)
.if !defined(TAGS) || ! ${TAGS:Mpackage=*}
TAGS+= package=${PACKAGE:Uruntime}
.endif
TAG_ARGS= -T ${TAGS:[*]:S/ /,/g}
.endif
2004-12-21 09:59:45 +00:00
.if defined(NO_SHARED) && (${NO_SHARED} != "no" && ${NO_SHARED} != "NO")
LDFLAGS+= -static
.endif
1994-05-30 19:09:18 +00:00
.if ${MK_DEBUG_FILES} != "no"
PROG_FULL=${PROG}.full
# Use ${DEBUGDIR} for base system debug files, else .debug subdirectory
.if defined(BINDIR) && (\
${BINDIR} == "/bin" ||\
${BINDIR:C%/libexec(/.*)?%/libexec%} == "/libexec" ||\
${BINDIR} == "/sbin" ||\
${BINDIR:C%/usr/(bin|bsdinstall|libexec|lpr|sendmail|sm.bin|sbin|tests)(/.*)?%/usr/bin%} == "/usr/bin" ||\
${BINDIR} == "/usr/lib" \
)
DEBUGFILEDIR= ${DEBUGDIR}${BINDIR}
.else
DEBUGFILEDIR?= ${BINDIR}/.debug
.endif
.if !exists(${DESTDIR}${DEBUGFILEDIR})
DEBUGMKDIR=
.endif
.else
PROG_FULL= ${PROG}
.endif
1994-05-30 19:09:18 +00:00
.if defined(PROG)
PROGNAME?= ${PROG}
1994-05-30 19:09:18 +00:00
.if defined(SRCS)
OBJS+= ${SRCS:N*.h:R:S/$/.o/g}
.if target(beforelinking)
beforelinking: ${OBJS}
${PROG_FULL}: beforelinking
.endif
${PROG_FULL}: ${OBJS}
.if defined(PROG_CXX)
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
${CXX:N${CCACHE_BIN}} ${CXXFLAGS:N-M*} ${LDFLAGS} -o ${.TARGET} \
${OBJS} ${LDADD}
.else
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
${CC:N${CCACHE_BIN}} ${CFLAGS:N-M*} ${LDFLAGS} -o ${.TARGET} ${OBJS} \
${LDADD}
.endif
.if ${MK_CTF} != "no"
${CTFMERGE} ${CTFFLAGS} -o ${.TARGET} ${OBJS}
.endif
.else # !defined(SRCS)
1994-05-30 19:09:18 +00:00
.if !target(${PROG})
.if defined(PROG_CXX)
SRCS= ${PROG}.cc
.else
SRCS= ${PROG}.c
.endif
1994-05-30 19:09:18 +00:00
# Always make an intermediate object file because:
# - it saves time rebuilding when only the library has changed
# - the name of the object gets put into the executable symbol table instead of
# the name of a variable temporary object.
# - it's useful to keep objects around for crunching.
OBJS+= ${PROG}.o
.if target(beforelinking)
beforelinking: ${OBJS}
${PROG_FULL}: beforelinking
.endif
${PROG_FULL}: ${OBJS}
.if defined(PROG_CXX)
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
${CXX:N${CCACHE_BIN}} ${CXXFLAGS:N-M*} ${LDFLAGS} -o ${.TARGET} \
${OBJS} ${LDADD}
.else
Add built-in ccache build support via WITH_CCACHE_BUILD option. ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not used to achieve a safe pseudo-incremental build. This is explained in more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the cache, but with a full cache saves 30-50% in build times. When combined with the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the resulting dependency file, which it does not do when using mkdep(1)/'CC -E'. Stats are provided at the end of this message. This removes the need to modify /etc/make.conf with the CC:= and CXX:= lines which conflicted with external compiler support [3] (causing the bootstrap compiler to not be built which lead to obscure failures [4]), incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid Clang errors with parenthesis, and did not work with META_MODE. The option name was picked to match the existing option in ports. This feature is available for both in-src and out-of-src builds that use /usr/share/mk. Linking, assembly compiles, and pre-processing avoid using ccache since it is only overhead. ccache does nothing special in these modes, although there is no harm in calling it for them. CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap compiler to hash the content of the compiler binary to determine if it should be a cache miss. For external compilers the 'mtime' option is used as it is more efficient and likely to be correct. Future work may optimize the 'content' check using the same checks as whether a bootstrap compiler is needed to be built. The CCACHE_CPP2 pessimization is currently default in our devel/ccache port due to Clang requiring it. Clang's -Wparentheses-equality, -Wtautological-compare, and -Wself-assign warnings do not mix well with compiling already-pre-processed code that may have expanded macros that trigger the warnings. GCC has so far not had this issue so it is allowed to disable the CCACHE_CPP2 default in our port. Sharing a cache between multiple checkouts, or systems, is explained in the ccache manual. Sharing a cache over NFS would likely not be worth it, but syncing cache directories between systems may be useful for an organization. There is also a memcached backend available [5]. Due to using an object directory outside of the source directory though you will need to ensure that both are in the same prefix and all users use the same layout. A possible working layout is as follows: Source: /some/prefix/src1 Source: /some/prefix/src2 Source: /some/prefix/src3 Objdir: /some/prefix/obj Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj' This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace all absolute paths to be relative. Using something like this is required due to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache adds into its hash for the object without CCACHE_BASEDIR. distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc. I have not personally tested this and assume it will not mix well with using the bootstrap compiler. The cache from buildworld can be reused in a subdir by first running 'make buildenv' (from r290424). Note that the cache is currently different depending on whether -j is used or not due to ccache enabling -fdiagnostics-color automatically if stderr is a TTY, which bmake only does if not using -j. The system I used for testing was: WITNESS Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log. The arc was fully populated with src tree files and ccache objects. RAM: 76GiB CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz 2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16 The WITH_FAST_DEPEND feature was used for comparison here as well to show the dramatic time savings with a full cache. buildworld: x buildworld-before + buildworld-ccache-empty * buildworld-ccache-full % buildworld-ccache-full-fastdep # buildworld-fastdep +-------------------------------------------------------------------------------+ |% * # +| |% * # +| |% * # xxx +| | |A | | A| | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 3744.13 3794.31 3752.25 3763.5633 26.935139 + 3 4519 4525.04 4520.73 4521.59 3.1104823 Difference at 95.0% confidence 758.027 +/- 43.4565 20.1412% +/- 1.15466% (Student's t, pooled s = 19.1726) * 3 1823.08 1827.2 1825.62 1825.3 2.0785572 Difference at 95.0% confidence -1938.26 +/- 43.298 -51.5007% +/- 1.15045% (Student's t, pooled s = 19.1026) % 3 1266.96 1279.37 1270.47 1272.2667 6.3971113 Difference at 95.0% confidence -2491.3 +/- 44.3704 -66.1952% +/- 1.17895% (Student's t, pooled s = 19.5758) # 3 3153.34 3155.16 3154.2 3154.2333 0.91045776 Difference at 95.0% confidence -609.33 +/- 43.1943 -16.1902% +/- 1.1477% (Student's t, pooled s = 19.0569) buildkernel: x buildkernel-before + buildkernel-ccache-empty * buildkernel-ccache-empty-fastdep % buildkernel-ccache-full # buildkernel-ccache-full-fastdep @ buildkernel-fastdep +-------------------------------------------------------------------------------+ |# @ % * | |# @ % * x + | |# @ % * xx ++| | MA | | MA| | A | | A | |A | | A | +-------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 3 571.57 573.94 571.79 572.43333 1.3094401 + 3 727.97 731.91 728.06 729.31333 2.2492295 Difference at 95.0% confidence 156.88 +/- 4.17129 27.4058% +/- 0.728695% (Student's t, pooled s = 1.84034) * 3 527.1 528.29 528.08 527.82333 0.63516402 Difference at 95.0% confidence -44.61 +/- 2.33254 -7.79305% +/- 0.407478% (Student's t, pooled s = 1.02909) % 3 400.4 401.05 400.62 400.69 0.3306055 Difference at 95.0% confidence -171.743 +/- 2.16453 -30.0023% +/- 0.378128% (Student's t, pooled s = 0.954969) # 3 201.94 203.34 202.28 202.52 0.73020545 Difference at 95.0% confidence -369.913 +/- 2.40293 -64.6212% +/- 0.419774% (Student's t, pooled s = 1.06015) @ 3 369.12 370.57 369.3 369.66333 0.79033748 Difference at 95.0% confidence -202.77 +/- 2.45131 -35.4225% +/- 0.428227% (Student's t, pooled s = 1.0815) [1] https://ccache.samba.org/performance.html [2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html [3] https://reviews.freebsd.org/D3484 [5] https://github.com/jrosdahl/ccache/pull/30 PR: 182944 [4] MFC after: 3 weeks Sponsored by: EMC / Isilon Storage Division Relnotes: yes
2015-11-08 00:50:18 +00:00
${CC:N${CCACHE_BIN}} ${CFLAGS:N-M*} ${LDFLAGS} -o ${.TARGET} ${OBJS} \
${LDADD}
1994-05-30 19:09:18 +00:00
.endif
.if ${MK_CTF} != "no"
${CTFMERGE} ${CTFFLAGS} -o ${.TARGET} ${OBJS}
.endif
.endif # !target(${PROG})
.endif # !defined(SRCS)
1994-05-30 19:09:18 +00:00
.if ${MK_DEBUG_FILES} != "no"
${PROG}: ${PROG_FULL} ${PROGNAME}.debug
${OBJCOPY} --strip-debug --add-gnu-debuglink=${PROGNAME}.debug \
${PROG_FULL} ${.TARGET}
${PROGNAME}.debug: ${PROG_FULL}
${OBJCOPY} --only-keep-debug ${PROG_FULL} ${.TARGET}
.endif
.if defined(LLVM_LINK)
# LLVM bitcode / textual IR representations of the program
BCOBJS= ${OBJS:.o=.bco}
LLOBJS= ${OBJS:.o=.llo}
${PROG_FULL}.bc: ${BCOBJS}
${LLVM_LINK} -o ${.TARGET} ${BCOBJS}
${PROG_FULL}.ll: ${LLOBJS}
${LLVM_LINK} -S -o ${.TARGET} ${LLOBJS}
.endif # defined(LLVM_LINK)
.if ${MK_MAN} != "no" && !defined(MAN) && \
!defined(MAN1) && !defined(MAN2) && !defined(MAN3) && \
1994-05-30 19:09:18 +00:00
!defined(MAN4) && !defined(MAN5) && !defined(MAN6) && \
!defined(MAN7) && !defined(MAN8) && !defined(MAN9)
MAN= ${PROG}.1
MAN1= ${MAN}
1994-05-30 19:09:18 +00:00
.endif
.endif # defined(PROG)
1994-05-30 19:09:18 +00:00
.if defined(_SKIP_BUILD)
all:
.else
all: ${PROG} ${SCRIPTS}
.if ${MK_MAN} != "no"
all: all-man
.endif
.endif
1994-05-30 19:09:18 +00:00
.if defined(PROG)
CLEANFILES+= ${PROG} ${PROG}.bc ${PROG}.ll
.if ${MK_DEBUG_FILES} != "no"
CLEANFILES+= ${PROG_FULL} ${PROG_FULL}.bc ${PROGNAME}.debug ${PROG_FULL}.ll
.endif
.endif
.if defined(OBJS)
CLEANFILES+= ${OBJS} ${BCOBJS} ${LLOBJS}
.endif
1994-05-30 19:09:18 +00:00
.include <bsd.libnames.mk>
.if defined(PROG)
.if !defined(NO_EXTRADEPEND)
_EXTRADEPEND:
.if defined(LDFLAGS) && !empty(LDFLAGS:M-nostdlib)
.if defined(DPADD) && !empty(DPADD)
echo ${PROG_FULL}: ${DPADD} >> ${DEPENDFILE}
.endif
.else
echo ${PROG_FULL}: ${LIBC} ${DPADD} >> ${DEPENDFILE}
.if defined(PROG_CXX)
.if ${COMPILER_TYPE} == "clang" && empty(CXXFLAGS:M-stdlib=libstdc++)
echo ${PROG_FULL}: ${LIBCPLUSPLUS} >> ${DEPENDFILE}
.else
echo ${PROG_FULL}: ${LIBSTDCPLUSPLUS} >> ${DEPENDFILE}
.endif
.endif
.endif
.endif # !defined(NO_EXTRADEPEND)
.endif
1994-05-30 19:09:18 +00:00
.if !target(install)
2004-11-12 13:24:34 +00:00
.if defined(PRECIOUSPROG)
2004-12-21 09:53:14 +00:00
.if !defined(NO_FSCHG)
INSTALLFLAGS+= -fschg
.endif
2004-11-12 13:24:34 +00:00
INSTALLFLAGS+= -S
.endif
_INSTALLFLAGS:= ${INSTALLFLAGS}
.for ie in ${INSTALLFLAGS_EDIT}
_INSTALLFLAGS:= ${_INSTALLFLAGS${ie}}
.endfor
.if !target(realinstall) && !defined(INTERNALPROG)
realinstall: _proginstall
.ORDER: beforeinstall _proginstall
_proginstall:
1994-05-30 19:09:18 +00:00
.if defined(PROG)
${INSTALL} ${TAG_ARGS} ${STRIP} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \
${_INSTALLFLAGS} ${PROG} ${DESTDIR}${BINDIR}/${PROGNAME}
.if ${MK_DEBUG_FILES} != "no"
.if defined(DEBUGMKDIR)
${INSTALL} ${TAG_ARGS:D${TAG_ARGS},debug} -d ${DESTDIR}${DEBUGFILEDIR}/
.endif
${INSTALL} ${TAG_ARGS:D${TAG_ARGS},debug} -o ${BINOWN} -g ${BINGRP} -m ${DEBUGMODE} \
${PROGNAME}.debug ${DESTDIR}${DEBUGFILEDIR}/${PROGNAME}.debug
.endif
.endif
.endif # !target(realinstall)
1994-05-30 19:09:18 +00:00
.if defined(SCRIPTS) && !empty(SCRIPTS)
realinstall: _scriptsinstall
.ORDER: beforeinstall _scriptsinstall
SCRIPTSDIR?= ${BINDIR}
SCRIPTSOWN?= ${BINOWN}
SCRIPTSGRP?= ${BINGRP}
SCRIPTSMODE?= ${BINMODE}
STAGE_AS_SETS+= scripts
stage_as.scripts: ${SCRIPTS}
FLAGS.stage_as.scripts= -m ${SCRIPTSMODE}
STAGE_FILES_DIR.scripts= ${STAGE_OBJTOP}
.for script in ${SCRIPTS}
.if defined(SCRIPTSNAME)
SCRIPTSNAME_${script:T}?= ${SCRIPTSNAME}
.else
SCRIPTSNAME_${script:T}?= ${script:T:R}
.endif
SCRIPTSDIR_${script:T}?= ${SCRIPTSDIR}
SCRIPTSOWN_${script:T}?= ${SCRIPTSOWN}
SCRIPTSGRP_${script:T}?= ${SCRIPTSGRP}
SCRIPTSMODE_${script:T}?= ${SCRIPTSMODE}
STAGE_AS_${script:T}= ${SCRIPTSDIR_${script:T}}/${SCRIPTSNAME_${script:T}}
_scriptsinstall: _SCRIPTSINS_${script:T}
_SCRIPTSINS_${script:T}: ${script}
${INSTALL} ${TAG_ARGS} -o ${SCRIPTSOWN_${.ALLSRC:T}} \
-g ${SCRIPTSGRP_${.ALLSRC:T}} -m ${SCRIPTSMODE_${.ALLSRC:T}} \
${.ALLSRC} \
${DESTDIR}${SCRIPTSDIR_${.ALLSRC:T}}/${SCRIPTSNAME_${.ALLSRC:T}}
.endfor
.endif
NLSNAME?= ${PROG}
.include <bsd.nls.mk>
.include <bsd.confs.mk>
.include <bsd.files.mk>
.include <bsd.incs.mk>
.include <bsd.links.mk>
.if ${MK_MAN} != "no"
realinstall: maninstall
.ORDER: beforeinstall maninstall
.endif
Fix most cases of bsd.progs.mk running duplicate or missing commands. This mostly fixes an interaction with bsd.test.mk with PROGS and SCRIPTS. This was most notable with 'make clean' and 'make install', which r281055 and r272055 attempted to address but were inadequate. It also addresses similar issues in bsd.progs.mk when not using bsd.test.mk. This also fixes cases of NOT running commands in the parent when using bsd.progs.mk: - 'make clean' was not run for the main process for Makefiles which had both FILES and SUBDIR but no PROGS or SCRIPTS. This usually was just a leftover Kyuafile.auto. One such example is usr.bin/bmake/tests/sysmk/t1/2. - 'make obj' was not running in the current directory with bsd.test.mk due to early inclusion of bsd.subdir.mk. This was not really a problem due to the SUBDIRS using 'mkdir -p' for their objdirs. There were subtle bugs causing this wrong behavior: 1. bsd.progs.mk needs to set SCRIPTS to empty when recursing to avoid the sub-makes from installing, cleaning or building the SCRIPTS; only the parent make should be doing this. r281055 effectively did the same but wasn't enough. 2. CLEANFILES may contain (especially from *.test.mk) files which only the parent should clean, such as from FILES and SCRIPTS. To resolve sub-makes also cleaning these, reset CLEANFILES and CLEANDIRS in the children before including bsd.prog.mk. A tempting alternative would be to only handle CLEANFILES in the parent but then the child bsd.prog.mk CLEANFILES of per-PROGS wouldn't be setup. 3. bsd.subdir.mk was included too soon in bsd.test.mk. It needs to be included after bsd.prog.mk as the SCRIPTS logic is short-circuitted if 'install:' is already defined (which bsd.subdir.mk does). There is actually no need to include bsd.subdir.mk from bsd.test.mk as bsd.prog.mk and bsd.obj.mk will do so in the proper order. The description in r257095 covers this for FILES and was fixed differently, though changing the handling of target(install) in bsd.prog.mk may make sense after more research. 4. bsd.progs.mk had extra logic to handle recursing SCRIPTS if PROGS was empty, which isn't its business to be doing. SCRIPTS is handled fine by bsd.prog.mk. This mostly reverts and reworks the fix in r259209 and partially reverts r272055. 5. bsd.progs.mk has no need to depend 'all:' on SCRIPTS and FILES. These are handled by bsd.prog.mk/bsd.files.mk fine. This also partially reverts r272055. 6. bsd.progs.mk was not drop-in safe for bsd.prog.mk. Move the PROGS check from r273186 to allow it to be used safely. Specific tested cases: SCRIPTS:no PROGS:no FILES:yes SUBDIR:yes usr.bin/bmake/tests/sysmk/t1/2 SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no usr.bin/bmake/tests/sysmk/t1/2/1 SCRIPTS:yes PROGS:yes FILES:yes SUBDIR:yes lib/libthr/tests SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no usr.bin/yacc/tests libexec/atf/atf-sh/tests A full buildworld/installworld/clean comparison with mtree was also done. The only relevant difference was the new fixed behavior of removing Kyuafile.auto from the objdir in 'clean'. Converting SCRIPTS to be a special case FILES group will make this less fragile and is being explored. One known remaining issue is 'cleandepend' removing the tags files for every recursive call. Note that the 'make clean' command runs for the CURDIR last, which can make it appear to run multiple times when cleaning in tests/, but each command is for a SUBDIR returning up the chain. This is purely bsd.subdir.mk behavior. PR: 191055 PR: 191955 MFC after: 2 weeks Sponsored by: EMC / Isilon Storage Division
2015-09-23 23:20:49 +00:00
.endif # !target(install)
1994-05-30 19:09:18 +00:00
.if !target(lint)
lint: ${SRCS:M*.c}
1994-05-30 19:09:18 +00:00
.if defined(PROG)
${LINT} ${LINTFLAGS} ${CFLAGS:M-[DIU]*} ${.ALLSRC}
1994-05-30 19:09:18 +00:00
.endif
.endif
.if ${MK_MAN} != "no"
1994-05-30 19:09:18 +00:00
.include <bsd.man.mk>
.endif
.if defined(PROG)
OBJS_DEPEND_GUESS+= ${SRCS:M*.h}
.endif
.include <bsd.dep.mk>
.include <bsd.clang-analyze.mk>
.include <bsd.obj.mk>
.include <bsd.sys.mk>