Commit Graph

119 Commits

Author SHA1 Message Date
Bryan Drewery
80f9a973ef FAST_DEPEND: Still need to remove DEPENDFILE when rebuilding.
This avoids _EXTRADEPEND adding duplicate entries.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 20:28:45 +00:00
Bryan Drewery
b9cd412e7b FAST_DEPEND: Always run depend via beforebuild which removes many hacks.
This will generate dependencies rather than depending on the previous behavior
of depending on the guessed OBJS: *.h dependecies or a user running
'make depend'.

Experimentation showed that depending only on headers was not enough and
prone to .ORDER errors.  Downstream users may also have added
dependencies into beforedepend or afterdepend targets.  The safest way to
ensure dependencies are generated before build is to run 'make depend'
beforehand rather than just depending on DPSRCS+SRCS.

Note that the OBJS_DEPEND_GUESS mechanism (a.k.a .if !exists(.depend) then
foo.o: *.h) is still useful as it improves incremental builds with missing
.depend.* files and allows 'make foo.o' to usually work, while this
'beforebuild: depend' ensures that the build will always find all dependencies.
The 'make foo.o' case has no means of a 'beforebuild' hook.

This also removes several hacks in the DIRDEPS_BUILD:
- NO_INSTALL_INCLUDES is no longer needed as it mostly was to work around
  .ORDER problems with building the needed headers early.
- DIRDEPS_BUILD: It is no longer necesarry to track "local dependencies" in
  Makefile.depend.

  These were only in Makefile.depend for 'clean builds' since nothing would
  generate the files due to skipping 'make depend' and early dependency
  bugs that have been fixed, such as adding headers into SRCS for the
  OBJS_DEPEND_GUESS mechanism.  Normally if a .depend file does not exist then
  a dependency is added by bsd.lib.mk/bsd.prog.mk from OBJS: *.h.  However,
  meta.autodep.mk creates a .depend file from created meta files and inserts
  that into Makefile.depend.  It also only tracks *.[ch] files though which can
  miss some dependencies that are hooked into 'make depend'.  This .depend
  that is created then breaks incremental builds due to the !exists(.depend)
  checks for OBJS_DEPEND_GUESS.  The goal was to skip 'make depend' yet it only
  really works the first time.  After that files are not generated as expected,
  which r288966 tried to address but was using buildfiles: rather than
  beforebuild: and was reverted in r291725.  As noted previously,
  depending only on headers in beforebuild: would create .ORDER errors
  in some cases.

  meta.autodep.mk is still used to generate Makefile.depend though via:
    gendirdeps: Makefile.depend
    .END: gendirdeps

  This commit allows removing all of the "local dependencies" in
  Makefile.depend which cuts down on churn and removes some of the
  arch-dependent Makefile.depend files.

  The "local dependencies" were also problematic for bootstrapping.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:19:18 +00:00
Bryan Drewery
70ca9ec4b9 Hook the meta/nofilemon build into using FAST_DEPEND.
FAST_DEPEND is intended to be the "skip 'make depend' and mkdep"
feature.  Since DIRDEPS_BUILD does this already with some of its own
hacks, and filemon doesn't need this, and nofilemon does, teach it how
to handle each of these cases.

In meta+filemon mode filemon will handle dependencies itself via the
meta mode logic in bmake.  We still want to set MK_FAST_DEPEND=yes to
enable some logic that indicates that 'make depend' is skipped in the
traditional sense.  The actual .depend.* files will be skipped.

When nofilemon is set though we still need to track and generate dependencies.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:19:13 +00:00
Bryan Drewery
68add21344 FAST_DEPEND: Don't waste time generating an empty .depend file.
The .depend file will still be generated if _EXTRADEPEND is used.  The target
is kept with a dependency on DPSRCS though so that 'make depend' will generate
all files.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:19:09 +00:00
Bryan Drewery
d7beff082b FAST_DEPEND: Rework how guessed dependencies are handled.
Rather than depend on .depend not existing, check the actual
.depend.OBJ file that will be used for that object.  If it doesn't
exist then use the guessed dependencies.

FAST_DEPEND may never have a .depend file.  Not having one means all of the
previous logic would over-depend all object files on all headers which is not
what we wanted.  It also means that if a .depend is generated before a build
is done for _EXTRADEPEND (such as for PROG or LIB) then all of these
dependencies would not be used since the .depend wasn't generated from mkdep
and the real .depend.* files are not generated until the build.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:19:05 +00:00
Bryan Drewery
3d437ee23f Follow-up r295667 with fixes for SRCS defined.
cleandepend should always remove CLEANDEPEND* if they are not empty,
but bsd.dep.mk should not add the tags entries unless SRCS is defined
as it did before.  The .depend file itself it still always removed
to avoid accidentally keeping a stale one around as done in r295666.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:18:55 +00:00
Bryan Drewery
9fb02a70d0 Avoid reading .depend.* in simple cases where not needed.
This will speed up some tree-walks with FAST_DEPEND which otherwise
would include length(SRCS) .depend files.

This also uses a trick suggested by sjg@ to still read them in when
specifying _V_READ_DEPEND=1 in the env/make args.

Sponsored by:	EMC / Isilon Storage Division
2016-02-19 00:41:24 +00:00
Bryan Drewery
ae038fc336 Add CLEANDEPEND[FILES|DIRS] and simplify the tags cleanup to use it.
Sponsored by:	EMC / Isilon Storage Division
2016-02-16 19:11:17 +00:00
Bryan Drewery
b601b230d8 Always remove .depend.* regardless of WITH_FAST_DEPEND in case of changing
which is used.

Sponsored by:	 EMC / Isilon Storage Division
2016-02-16 19:02:04 +00:00
Bryan Drewery
3b1e60a40a Remove temporary hack from r294370 for SSH upgrades.
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-02-16 19:00:22 +00:00
Bryan Drewery
2cf60b3ce2 Fix -include .depend hack from r294370 for headers not in .PATH.
This hack will be removed in a few weeks.  It is here to fix incremental
builds of SSH between r291941 and r294370.

Reported by:	jmallett
MFC after:	1 day
Sponsored by:	EMC / Isilon Storage Division
2016-01-28 18:57:47 +00:00
Bryan Drewery
5444d429b9 Revert yacc dependency back to pre-r241298.
Several attempts to fix this logic was done after r241298, which were
all reverted, yet this change was not.

The .h file does not depend on the .c file, so do not impose such a
dependency on it.  They are generated by the same command but do not
depend on each other.  Restore the .ORDER which should handle parallel build
issues.  This fixes an actual bug where the .h file is not recreated
when missing [1].  For example:
  cd lib/libc
  make cleanobj
  make nsparser.h
  rm nsparser.h
  make nsparser.h # will not rebuild nsparser.h

I have been trying to track down a build problem where nsparser.h is
missing when nslexer.o is built.  It is possible this is related.

Reported by:	bde [1]
		https://lists.freebsd.org/pipermail/svn-src-all/2012-October/059481.html
		https://lists.freebsd.org/pipermail/svn-src-all/2012-October/060038.html
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
2016-01-27 01:33:26 +00:00
Bryan Drewery
369737d7f0 Fix DIRDEPS_BUILD after r294752.
DIRDEPS_BUILD does not yet support PROGS having their own dependency
file.

Overriding .MAKE.DEPENDFILE here causes major problems with the meta
mode logic since it creates the Makefile.depend as '.depend' resulting
in infinite loops in make due to dirdeps.mk including .depend endlessly.

X-MFC-With:	r294752
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-01-27 01:24:05 +00:00
Bryan Drewery
fc2c2cd9e3 Fix PROGS not reading .depend files after r284288 by making DEPENDFILE work.
We have had this user-modifable DEPENDFILE variable forever that does nothing
relevant for the user since fmake always used '.depend'.  Bmake
introduced the .MAKE.DEPENDFILE variable that can be modified to change
the name of '.depend'.

Prior to r284288, bsd.progs.mk was setting .MAKE.DEPENDFILE to allow
working incremental builds.  This was modified most likely to not
conflict with the META MODE handling of .MAKE.DEPENDFILE as it has a lot
more special logic for that variable.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-01-25 22:29:44 +00:00
Bryan Drewery
cfa5f2be13 Remove unneeded beforebuild hacks for dtrace probes.
This still allows skipping 'make depend'.  Headers in SRCS generate
properly.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-01-25 22:29:41 +00:00
Bryan Drewery
a298c2797c Fix incremental build of dtrace probes.
Currently dtrace(1) -Go does not properly rebuild the target if it
exists.  It results in missing symbols.

  dtrace -C -x nolibs -G -o usdt.o -s /root/git/freebsd/cddl/contrib/opensolaris/cmd/dtrace/test/tst/common/json/usdt.d tst.usdt.o
  dtrace: target object (usdt.o) already exists. Please remove the target
  dtrace: object and rebuild all the source objects if you wish to run the DTrace
  dtrace: linking process again
  cc -O2 -pipe -O0 -g -I/root/git/freebsd/cddl/usr.sbin/dtrace/tests/common/json -std=gnu99 -fstack-protector-strong -Qunused-arguments  -o tst.usdt.exe.full tst.usdt.o usdt.o
  tst.usdt.o: In function `main':
  /root/git/freebsd/cddl/contrib/opensolaris/cmd/dtrace/test/tst/common/json/tst.usdt.c:56: undefined reference to `__dtrace_bunyan_fake___log__debug'
  /root/git/freebsd/cddl/contrib/opensolaris/cmd/dtrace/test/tst/common/json/tst.usdt.c:60: undefined reference to `__dtrace_bunyan_fake___log__debug'
  cc: error: linker command failed with exit code 1 (use -v to see invocation)
  *** [tst.usdt.exe.full] Error code 1

This is a consequence of r212358.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-01-25 22:29:32 +00:00
Bryan Drewery
473bcb0163 mkdep: Fix -include not being added for .depend tracking.
This fixes incremental build of OpenSSH after the recent upgrade.

For example, in secure/lib/libssh, -include ssh_namespace.h is used on
all files.  This is not tracked in the .depend file though due to
MKDEP_CFLAGS not including it.  The ssh example was broken in r291941
when not using FAST_DEPEND due to the .depend bug.  FAST_DEPEND was not
affected by this because it generates dependencies at compile time and
thus sees the -include.

This ugly make syntax could be simpler for bmake by using :tW but
fmake-compatible syntax is used since this needs to be MFC'd all the way
to stable/9.

Also add a temporary hack to workaround existing checkouts building
incrementally with a .depend file not having these headers.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-01-20 01:40:18 +00:00
Bryan Drewery
d3b887fd0d FAST_DEPEND: Still use if filemon is not used.
If filemon is used then there is no need to generate dependency files during
compilation as the .meta files will achieve the same result.

This is a temporary solution until FAST_DEPEND is default.  Once that is
default there will be an option to disable dependency generation entirely
as it is only useful if an incremental build is planned, thus META_MODE+filemon
can enable that option to short-circuit all FAST_DEPEND-related logic.

Sponsored by:	EMC / Isilon Storage Division
2016-01-19 22:41:58 +00:00
Bryan Drewery
0b6ba3f22a Define .MAKE.MODE to normal to avoid the need for :U later.
Sponsored by:	EMC / Isilon Storage Division
2016-01-19 22:41:44 +00:00
Bryan Drewery
9f286e8295 FAST_DEPEND: Rework optimization for r290524.
The .MAKEFLAGS check inside of the .for loop is extremely slow for some
reason.  Just moving it out of the loop trimmed -V lookup time from 11
seconds to 1 second in the kernel obj directory.

Sponsored by:	EMC / Isilon Storage Division
2016-01-15 22:08:51 +00:00
Bryan Drewery
10f696a52d FAST_DEPEND: Only pass -MF if we care about the object being compiled.
This will save time generating dependency files that we didn't expect
due to cases where SRCS!=OBJS or for building custom targetted objects
in Makefiles that do not end up in the DEPENDOBJS list.

This uses a bmake trick to modify CFLAGS based on ${.TARGET}.  A
.PARSEDIR check is done for the sake of MFC safety.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-12-07 16:08:22 +00:00
Bryan Drewery
b9964333e0 FAST_DEPEND: Move handling code below yacc/lex/dtrace code that modified SRCS.
This fixes some of those newly added SRCS not having their depend files
included.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-12-07 16:08:19 +00:00
Bryan Drewery
69afbf0275 FAST_DEPEND: Only try to use dependencies from C/C++ SRCS as mkdep did.
Rather than try to guess at all of the OBJS variables just use SRCS
using the same patterns that mkdep does.  This also fixes a mistake
where dependencies were being generated with FAST_DEPEND when they were
not for mkdep.  This happens when OBJS!=SRCS as is the case in
gnu/lib/csu where SRCS has 1 file and OBJS has several other files that
does not even contain the 1 SRCS file.  Generally in these cases the
OBJS have custom dependencies defined in their Makefile.  If we generate
dependencies for those and then load a .depend file, then .IMPSRC may
contain duplicate sources and lead to errors such as:
  cc: error: cannot specify -o when generating multiple output files

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-12-07 16:08:16 +00:00
Bryan Drewery
6feede08e1 FAST_DEPEND: Actually use -MP from DEPEND_MP, completing r291554.
X-MFC-With:	r291554
MFC after:	1 week
Pointyhat to:	bdrewery
Sponsored by:	EMC / Isilon Storage Division
2015-12-01 15:50:35 +00:00
Bryan Drewery
b6725ecdb2 FAST_DEPEND: Workaround hacks in the wild that have '..' or '/' in OBJS.
This is mostly working around the converts/iconv port having '../ces/file.o'
in its OBJS list which resulted in '.depend../ces/file.o'.  Now it will have
'.depend.._ces_file.o'.

Other implementations have :T which would result in '.depend.file.o' here, but
that could lead to collisions.

X-MFC-With:	r291554
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-12-01 03:09:54 +00:00
Bryan Drewery
593e465913 FAST_DEPEND: Allow not using -MP by setting DEPEND_MP blank.
-MP creates empty targets for all dependency files, which can be useful when a
dependency is deleted from the file system.  This would otherwise cause an
error for "don't know how to build FOO" since the .depend file is included
with the dependency registered.

This is mostly a workaround for the misc/dahdi-kmod port using '::' for one of
its dependencies, while -MP uses just ':'.  This results in an 'Inconsistent
operator for' error.

X-MFC-With:	r290433
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-12-01 03:00:10 +00:00
Simon J. Gerraty
948f327ee4 Rename META_MODE option to DIRDEPS_BUILD
This allows META_FILES option to be renamed META_MODE.
Also add META_COOKIE_TOUCH for use in targets that can benefit
from a cookie when in meta mode.

Differential Revision:	https://reviews.freebsd.org/D4153
Reviewed by:	bdrewery
2015-11-14 03:24:48 +00:00
Bryan Drewery
b03ca5c3ec FAST_DEPEND: Fix building of wrong source files in some cases.
Similar to the original reason for these dependency hints to be added,
in r124637, the missing-dependency file case can lead to building of the
wrong source.

A clear example of this is in gnu/lib/libstdc++ where the .PATH contains
both contrib/gcc and contrib/libstdc++/src.

  contrib/gcc has a debug.c.
  contrib/libstdc++/src has a debug.cc.

  When building for the objects of debug.o, debug.So, and debug.po, it is
  ambiguous for which src file to use due to the suffix transformation
  rules, even though the proper one is listed first in .PATH.

  This was normally avoided due to these dependency hints for the initial
  build, and then mkdep would add an explicit 'debug.o: debug.cc'
  dependency into the .depend file.  WITH_FAST_DEPEND does not generate
  the .depend file with these, but puts them into .depend.debug.o instead.

Rather than extending the exists() check to each object's .depend.*
file, just enable the hint when when using WITH_FAST_DEPEND.  It fixes
the problem and seems to be safe enough to use since it is mapping SRCS
back to OBJS, rather than letting make make assumptions from OBJS to
SRCS.

A similar check mapping objects to headers is present in some mk files
but was not extended here for FAST_DEPEND since it has not yet been
found to be a problem.

X-MFC-With:	r290433
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-11-09 23:37:04 +00:00
Bryan Drewery
9160419c7a Add built-in ccache build support via WITH_CCACHE_BUILD option.
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build.  This is explained in
more detail upstream [1] [2].  It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times.  When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'.  Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports.  This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead.  ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss.  For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct.  Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it.  Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings.  GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual.  Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization.  There is also a memcached backend available [5].  Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout.  A
possible working layout is as follows:
  Source: /some/prefix/src1
  Source: /some/prefix/src2
  Source: /some/prefix/src3
  Objdir: /some/prefix/obj
  Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative.  Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files and ccache objects.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
  x buildworld-before
  + buildworld-ccache-empty
  * buildworld-ccache-full
  % buildworld-ccache-full-fastdep
  # buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |%            *                               #                                +|
  |%            *                               #                                +|
  |%            *                               #             xxx                +|
  |                                                           |A                  |
  |                                                                              A|
  |             A                                                                 |
  |A                                                                              |
  |                                             A                                 |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3          4519       4525.04       4520.73       4521.59     3.1104823
  Difference at 95.0% confidence
          758.027 +/- 43.4565
          20.1412% +/- 1.15466%
          (Student's t, pooled s = 19.1726)
  *   3       1823.08        1827.2       1825.62        1825.3     2.0785572
  Difference at 95.0% confidence
          -1938.26 +/- 43.298
          -51.5007% +/- 1.15045%
          (Student's t, pooled s = 19.1026)
  %   3       1266.96       1279.37       1270.47     1272.2667     6.3971113
  Difference at 95.0% confidence
          -2491.3 +/- 44.3704
          -66.1952% +/- 1.17895%
          (Student's t, pooled s = 19.5758)
  #   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-ccache-empty
  * buildkernel-ccache-empty-fastdep
  % buildkernel-ccache-full
  # buildkernel-ccache-full-fastdep
  @ buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |#                        @   %                  *                              |
  |#                        @   %                  *     x                      + |
  |#                        @   %                  *     xx                     ++|
  |                                                      MA                       |
  |                                                                             MA|
  |                                                A                              |
  |                             A                                                 |
  |A                                                                              |
  |                         A                                                     |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        727.97        731.91        728.06     729.31333     2.2492295
  Difference at 95.0% confidence
          156.88 +/- 4.17129
          27.4058% +/- 0.728695%
          (Student's t, pooled s = 1.84034)
  *   3         527.1        528.29        528.08     527.82333    0.63516402
  Difference at 95.0% confidence
          -44.61 +/- 2.33254
          -7.79305% +/- 0.407478%
          (Student's t, pooled s = 1.02909)
  %   3         400.4        401.05        400.62        400.69     0.3306055
  Difference at 95.0% confidence
          -171.743 +/- 2.16453
          -30.0023% +/- 0.378128%
          (Student's t, pooled s = 0.954969)
  #   3        201.94        203.34        202.28        202.52    0.73020545
  Difference at 95.0% confidence
          -369.913 +/- 2.40293
          -64.6212% +/- 0.419774%
          (Student's t, pooled s = 1.06015)
  @   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR:		182944 [4]
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
Relnotes:	yes
2015-11-08 00:50:18 +00:00
Bryan Drewery
dbced32af5 FAST_DEPEND: Don't enable when .MAKE.MODE=meta.
This is because the .meta files generated from filemon already contain a
list of all files read to generate the object.

X-MFC-With:	r290433
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-11-08 00:50:13 +00:00
Bryan Drewery
3cccdc9d58 FAST_DEPEND: Don't include depend files when using 'make -V'.
This is especially noticeable in the kernel obj directory since it
includes so many files.

X-MFC-With:	r290433
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-11-08 00:50:09 +00:00
Bryan Drewery
cf1eeb33be Add a FAST_DEPEND option, off by default, which speeds up the build significantly.
This speeds up buildworld by 16% on my system and buildkernel by 35%.

Rather than calling mkdep(1), which is just a wrapper around 'cc -E',
use the modern -MD -MT -MF flags to gather and generate dependencies during
compilation.  This flag was introduced in GCC "a long time ago", in GCC 3.0,
and is also supported by Clang.  (It appears that ICC also supports this but I
do not have access to test it).  This avoids running the preprocessor *twice*
for every build, in both 'make depend' and 'make all'.  This is especially
noticeable when using ccache since it does not cache preprocessor results from
mkdep(1) / 'cc -E', but still speeds up compilation with the -MD flags.

For 'make depend' a tree-walk is still done to ensure that all DPSRCS
are generated when expected, and that beforedepend/afterdepend and
_EXTRADEPEND are all still respected.  In time this may change but for now
I've been conservative.  The time for a tree-walk with -j combined with
SUBDIR_PARALLEL is not significant.  For example, it takes about 9 seconds
with -j15 to walk all of src/ for 'make depend' now on my system.

A .depend file is still generated with the various rules that apply to
the final target, or custom rules.  Otherwise there are now
per-built-object-file .depend files, such as .depend.filename.o.  These
are included directly by make rather than populating .depend with a loop
and .depend lines, which only added overhead to the now almost-NOP 'make
depend' phase.

Before this I experimented with having mkdep(1) called in parallel per-file.
While this improved the kernel and lib/libc 'make depend' phase, it resulted
in slower build times overall.

The -M flags are removed from CFLAGS when linking since they have no effect.

Enabling this by default, for src or out-of-src, can be done once more testing
has been done, such as a ports exp-run, and with more compilers.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_FAST_DEPEND=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

buildworld:
  x buildworld-before
  + buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |+                                                                              |
  |+                                                                              |
  |+                                                                       xx    x|
  |                                                                       |_MA___||
  |A                                                                              |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |+                                                                            x |
  |++                                                                           xx|
  |                                                                             A||
  |A|                                                                             |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

Sponsored by:	EMC / Isilon Storage Division
MFC after:	3 weeks
Relnotes:	yes
2015-11-06 04:45:29 +00:00
Simon J. Gerraty
f5374544f1 Building on fmake is no longer possible so removed tests for bmake. 2015-06-10 18:14:38 +00:00
Simon J. Gerraty
b17ff922d4 Merge from head 2015-05-26 21:52:57 +00:00
Mark Johnston
0c76c8aa8c Add a DTRACEFLAGS variable, which can be used to pass additional variables
to dtrace(1) invocations during a build. This change includes -C in the
default flags, which has dtrace(1) run input scripts through the
preprocessor. While here, sort the definitions of CP and CPP in sys.mk.

Differential Revision:	https://reviews.freebsd.org/D2204
Reviewed by:		imp, rpaulo (previous revision)
2015-04-08 02:43:05 +00:00
Mark Johnston
03a5f9f0fd Remove drti.o's dependency on libelf. This makes it possible to add DTrace
probes to userland programs and libraries without also needing to link
libelf.

dtrace -G places the __SUNW_dof symbol at the beginning of the DOF (DTrace
probe and provider metdata) section in the generated object file; drti.o
now just uses this symbol to locate the section. A complication occurs
when multiple dtrace-generated object files are linked together, since the
__SUNW_dof symbol defined in each file is global. This is handled by
using objcopy(1) to convert __SUNW_dof to a local symbol once drti.o has
been linked with the generated object file. Upstream, this is done using a
linker feature not present in GNU ld.

Differential Revision:	https://reviews.freebsd.org/D1757
Reviewed by:	rpaulo
MFC after:	1 month
Relnotes:	yes
2015-02-18 03:54:54 +00:00
Simon J. Gerraty
9268022b74 Merge from head@274682 2014-11-19 01:07:58 +00:00
Baptiste Daroussin
8447c44fe7 The dependencies are computed with CC even if sources are C++, when building
when building with an external gcc, we want to be able to pass the path to
the libc++ headers so dependencies are correctly computed for C++ source files.
Add a DEPFLAGS for that purpose

Reviewed by:	imp
2014-10-21 19:56:45 +00:00
Mark Johnston
e9083b2dc5 Fix a few small bugs in the DTrace USDT rules:
* anchor search strings appropriately,
* use .ALLSRC to pass the full path to the D script to dtrace(1),
* don't insert the auto-generated header into SRCS - it doesn't
  accomplish anything, and we end up having to remove it from OBJS anyway.

Reviewed by:		rpaulo
Differential Revision:	https://reviews.freebsd.org/D978
MFC after:		3 weeks
Sponsored by:		EMC / Isilon Storage Division
2014-10-21 04:30:00 +00:00
Mark Johnston
721dae3a52 Correct the way that libelf is linked when USDT is used. 2014-10-01 04:28:39 +00:00
Enji Cooper
41ae64f374 Remove many false positives with make checkdpadd
- Reduce DPADD and LDADD in checkdpadd to -l<foo>
- Skip over -Wl,[es]*-group because -Wl,--end-group and
  -Wl,--start-group might be required to properly link objects (see
  usr.bin/clang/lldb as an example)

This caveat has been present for a while with some components of
the build. However, these false positives were made more more apparent
after r269648.

Phabric: D635
Reviewed by: jmmv (an earlier version)
PR: 192730
MFC after: 2 weeks
2014-09-10 07:55:51 +00:00
Mark Johnston
5439a71566 Define _DTRACE_VERSION in sdt.h rather than setting it manually. This is
similar to what illumos does, and makes it easier to enable USDT probes
in third-party software that doesn't make use of the system makefiles.
2014-09-02 23:43:06 +00:00
Simon J. Gerraty
ee7b0571c2 Merge head from 7/28 2014-08-19 06:50:54 +00:00
Rui Paulo
9c851217b6 Make sure the DTrace header files are built before depend and before
the build starts.

This adds a new variable DHDRS that contains a list of all DTrace
header files.  Then, we use the beforedepend hook to make sure the
heaeder files are built.

Introduce a beforebuild dependency (from projects/bmake) based on
feedback from Simon J. Gerraty.  This lets us generate the header
files without running make depend.

Reviewed by:	sjg, imp
MFC after:	3 days
2014-08-13 01:27:51 +00:00
Rui Paulo
17b227229c Revert r269839.
This doesn't work like I expected and the if clause is never matched.
It turns out that the problem with the older make is unrelated.

MFC after:	3 days
2014-08-11 22:43:44 +00:00
Rui Paulo
8a152f97e7 Add .d files to the reverse matching pattern in bsd.dep.mk to account
for source file dependencies.

This follows the .y and .l behaviour.

MFC after:	3 days
2014-08-11 21:42:06 +00:00
Rui Paulo
6b2edf61bd Fix another typoe in the definition of DTrace SOBJS.
MFC after:	3 days
2014-08-11 21:14:08 +00:00
Rui Paulo
04c241ea2d bsd.dep.mk: use the empty() function with the check for .d files.
Seems to work better with the old make.

MFC after:	3 days
2014-08-11 20:44:57 +00:00
Rui Paulo
23e9784542 Fix an error in the dependency of the DTrace object file.
MFC after:	3 days
2014-08-11 20:41:14 +00:00
Rui Paulo
9b646f2533 Fix a few problems with the USDT probes:
* Include OBJDIR to make sure the generated file is found;
* Simplify the definition of OBJS;
* Add targets for shared objects and for profiled objects.

MFC after:	3 days
2014-08-10 06:43:40 +00:00