Commit Graph

32 Commits

Author SHA1 Message Date
Warner Losh
9ca5434380 Move the inclusion of bsd.cpu.mk from sys.mk to bsd.opts.mk. However,
for historical behavior that ports depends on, include it if we're
inside the ports tree.

Differential Review: https://reviews.freebsd.org/D4383
Ports Exp run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=205021
2015-12-11 05:39:44 +00:00
Ed Maste
015dfda6c6 Build and install userland .debug files by default
Debug data files are now built by default with 'make buildworld' and
installed with 'make installworld'. This facilitates debugging but
requires more disk space both during the build and for the installed
world. Debug files may be disabled by setting WITHOUT_DEBUG_FILES=yes
in src.conf(5).

Reviewed by:	bdrewery, eadler, vangyzen
Relnotes:	Yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D4018
2015-12-07 19:15:52 +00:00
Bryan Drewery
9160419c7a Add built-in ccache build support via WITH_CCACHE_BUILD option.
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build.  This is explained in
more detail upstream [1] [2].  It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times.  When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'.  Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports.  This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead.  ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss.  For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct.  Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it.  Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings.  GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual.  Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization.  There is also a memcached backend available [5].  Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout.  A
possible working layout is as follows:
  Source: /some/prefix/src1
  Source: /some/prefix/src2
  Source: /some/prefix/src3
  Objdir: /some/prefix/obj
  Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative.  Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files and ccache objects.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
  x buildworld-before
  + buildworld-ccache-empty
  * buildworld-ccache-full
  % buildworld-ccache-full-fastdep
  # buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |%            *                               #                                +|
  |%            *                               #                                +|
  |%            *                               #             xxx                +|
  |                                                           |A                  |
  |                                                                              A|
  |             A                                                                 |
  |A                                                                              |
  |                                             A                                 |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3          4519       4525.04       4520.73       4521.59     3.1104823
  Difference at 95.0% confidence
          758.027 +/- 43.4565
          20.1412% +/- 1.15466%
          (Student's t, pooled s = 19.1726)
  *   3       1823.08        1827.2       1825.62        1825.3     2.0785572
  Difference at 95.0% confidence
          -1938.26 +/- 43.298
          -51.5007% +/- 1.15045%
          (Student's t, pooled s = 19.1026)
  %   3       1266.96       1279.37       1270.47     1272.2667     6.3971113
  Difference at 95.0% confidence
          -2491.3 +/- 44.3704
          -66.1952% +/- 1.17895%
          (Student's t, pooled s = 19.5758)
  #   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-ccache-empty
  * buildkernel-ccache-empty-fastdep
  % buildkernel-ccache-full
  # buildkernel-ccache-full-fastdep
  @ buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |#                        @   %                  *                              |
  |#                        @   %                  *     x                      + |
  |#                        @   %                  *     xx                     ++|
  |                                                      MA                       |
  |                                                                             MA|
  |                                                A                              |
  |                             A                                                 |
  |A                                                                              |
  |                         A                                                     |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        727.97        731.91        728.06     729.31333     2.2492295
  Difference at 95.0% confidence
          156.88 +/- 4.17129
          27.4058% +/- 0.728695%
          (Student's t, pooled s = 1.84034)
  *   3         527.1        528.29        528.08     527.82333    0.63516402
  Difference at 95.0% confidence
          -44.61 +/- 2.33254
          -7.79305% +/- 0.407478%
          (Student's t, pooled s = 1.02909)
  %   3         400.4        401.05        400.62        400.69     0.3306055
  Difference at 95.0% confidence
          -171.743 +/- 2.16453
          -30.0023% +/- 0.378128%
          (Student's t, pooled s = 0.954969)
  #   3        201.94        203.34        202.28        202.52    0.73020545
  Difference at 95.0% confidence
          -369.913 +/- 2.40293
          -64.6212% +/- 0.419774%
          (Student's t, pooled s = 1.06015)
  @   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR:		182944 [4]
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
Relnotes:	yes
2015-11-08 00:50:18 +00:00
Bryan Drewery
cf1eeb33be Add a FAST_DEPEND option, off by default, which speeds up the build significantly.
This speeds up buildworld by 16% on my system and buildkernel by 35%.

Rather than calling mkdep(1), which is just a wrapper around 'cc -E',
use the modern -MD -MT -MF flags to gather and generate dependencies during
compilation.  This flag was introduced in GCC "a long time ago", in GCC 3.0,
and is also supported by Clang.  (It appears that ICC also supports this but I
do not have access to test it).  This avoids running the preprocessor *twice*
for every build, in both 'make depend' and 'make all'.  This is especially
noticeable when using ccache since it does not cache preprocessor results from
mkdep(1) / 'cc -E', but still speeds up compilation with the -MD flags.

For 'make depend' a tree-walk is still done to ensure that all DPSRCS
are generated when expected, and that beforedepend/afterdepend and
_EXTRADEPEND are all still respected.  In time this may change but for now
I've been conservative.  The time for a tree-walk with -j combined with
SUBDIR_PARALLEL is not significant.  For example, it takes about 9 seconds
with -j15 to walk all of src/ for 'make depend' now on my system.

A .depend file is still generated with the various rules that apply to
the final target, or custom rules.  Otherwise there are now
per-built-object-file .depend files, such as .depend.filename.o.  These
are included directly by make rather than populating .depend with a loop
and .depend lines, which only added overhead to the now almost-NOP 'make
depend' phase.

Before this I experimented with having mkdep(1) called in parallel per-file.
While this improved the kernel and lib/libc 'make depend' phase, it resulted
in slower build times overall.

The -M flags are removed from CFLAGS when linking since they have no effect.

Enabling this by default, for src or out-of-src, can be done once more testing
has been done, such as a ports exp-run, and with more compilers.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_FAST_DEPEND=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

buildworld:
  x buildworld-before
  + buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |+                                                                              |
  |+                                                                              |
  |+                                                                       xx    x|
  |                                                                       |_MA___||
  |A                                                                              |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |+                                                                            x |
  |++                                                                           xx|
  |                                                                             A||
  |A|                                                                             |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

Sponsored by:	EMC / Isilon Storage Division
MFC after:	3 weeks
Relnotes:	yes
2015-11-06 04:45:29 +00:00
Simon J. Gerraty
ccfb965433 Add META_MODE support.
Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision:       D2796
Reviewed by: brooks imp
2015-06-13 19:20:56 +00:00
Simon J. Gerraty
1e34437637 Add dependent option STAGING_MAN/STAGING
Use a single mlinks set, since some libs (eg. libc) do cross section
links.
2015-06-04 19:11:56 +00:00
Simon J. Gerraty
b17ff922d4 Merge from head 2015-05-26 21:52:57 +00:00
Baptiste Daroussin
bec133a78d Remove leftovers from r276551 2015-01-02 18:54:55 +00:00
Warner Losh
1fc864601c Fix typo in comments.
Noticed by: brooks@
2014-12-12 18:54:31 +00:00
Simon J. Gerraty
9268022b74 Merge from head@274682 2014-11-19 01:07:58 +00:00
Warner Losh
590461a4b8 Acutally, replaces NO_INCS with WITHOUT_INCLUDES and make
WITHOUT_TOOLCHAIN imply it instead.

Sponsored by: Netflix
2014-11-18 17:06:44 +00:00
Bryan Drewery
5608fd23c2 Revert r267233 for now. PIE support needs to be reworked.
1. 50+% of NO_PIE use is fixed by adding -fPIC to INTERNALLIB and other
   build-only utility libraries.
2. Another 40% is fixed by generating _pic.a variants of various libraries.
3. Some of the NO_PIE use is a bit absurd as it is disabling PIE (and ASLR)
   where it never would work anyhow, such as csu or loader. This suggests
   there may be better ways of adding support to the tree. Many of these
   cases can be fixed such that -fPIE will work but there is really no
   reason to have it in those cases.
4. Some of the uses are working around hacks done to some Makefiles that are
   really building libraries but have been using bsd.prog.mk because the code
   is cleaner. Had they been using bsd.lib.mk then NO_PIE would not have
   been needed.

We likely do want to enable PIE by default (opt-out) for non-tree consumers
(such as ports). For in-tree though we probably want to only enable PIE
(opt-in) for common attack targets such as remote service daemons and setuid
utilities. This is also a great performance compromise since ASLR is expected
to reduce performance. As such it does not make sense to enable it in all
utilities such as ls(1) that have little benefit to having it enabled.

Reported by:	kib
2014-08-19 15:04:32 +00:00
Simon J. Gerraty
ee7b0571c2 Merge head from 7/28 2014-08-19 06:50:54 +00:00
Baptiste Daroussin
f471720995 The GNU texinfo and GNU info pages are not built and installed
anymore, WITH_INFO knob has been added to allow to built and install
them again.

Reviewed by:	imp
2014-07-10 15:05:41 +00:00
Bryan Drewery
864c53ead8 In preparation for ASLR [1] support add WITH_PIE to support building with -fPIE.
This is currently an opt-in build flag. Once ASLR support is ready and stable
it should changed to opt-out and be enabled by default along with ASLR.

Each application Makefile uses opt-out to ensure that ASLR will be enabled by
default in new directories when the system is compiled with PIE/ASLR. [2]

Mark known build failures as NO_PIE for now.

The only known runtime failure was rtld.

[1] http://www.bsdcan.org/2014/schedule/events/452.en.html
Submitted by:		Shawn Webb <lattera@gmail.com>
Discussed between:	des@ and Shawn Webb [2]
2014-06-08 17:29:31 +00:00
Warner Losh
e23dd3b2d2 Better documentation for why KERBEROS and OPENSSH are here, and where
to look for all the other options that used to be here (well, in
bsd.own.mk)

Suggested by: des@
2014-05-27 16:30:54 +00:00
Warner Losh
7ecf742eeb Move DOCCOMPRESS to MK variable. 2014-05-10 16:38:54 +00:00
Warner Losh
ab72ce6b1b Migrate NO_WARN to MK_WARN. Support legacy NO_WARN usage. Remove a
check for EARLY_BUILD because it isn't necessary (MK_WARN=no will
always be defined for that).
2014-05-10 16:37:53 +00:00
Simon J. Gerraty
cc3f4b9965 Merge from head 2014-05-08 23:54:15 +00:00
Warner Losh
5d420b97e6 All the NO_foo options processed in src.opts.mk are really bsd.opts.mk
options, so move their processing there. This fixes issues with
Makefiles that define NO_MAN=t and only inlcude bsd.*.mk files. A few
ports fell into this category, and they should be fixed by this change.
Also, for now, disable the warning about NO_foo being deprecated. More
work is needed than anticipated before we can do that, so kill the
noise for now.
2014-05-07 18:14:56 +00:00
Warner Losh
c6063d0da8 Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.
2014-05-06 04:22:01 +00:00
Warner Losh
5e84b765f4 Move the /usr/src specific options to src.opts.mk. Move inclusion of
/etc/src.conf to this file as well. Now, it will only affect builds of
/usr/src and not others that use the bsd.*.mk files. Specifically
don't install src.opts.mk so we can catch when it 'leaks' into
bsd.*.mk again and have there be errors when this happens. Future
commits will move to including src.opts.mk instead of bsd.own.mk when
all that's needed is one of the MK_FOO options from src.opts.mk.
Future options should be placed here, unless they directly affect a
bsd.*.mk file, in which case they should be placed in bsd.opts.mk.
2014-05-06 04:21:48 +00:00
Warner Losh
6f9f01a2d8 NLS is used by the build system. 2014-04-29 14:52:39 +00:00
Simon J. Gerraty
3b8f084595 Merge head 2014-04-28 07:50:45 +00:00
Warner Losh
7061d0855c Split up the options into those used only by bsd.mk.* or sys.mk and
the rest. Add note of caution for the surprises that popped up. MK_SSP
is the only one used by sys.mk, and it likely needs even extra
attention.
2014-04-25 19:25:53 +00:00
Warner Losh
c772d4925c Now that all the NO_foo expressed in this loop have been removed from
the tree, start warning that NO_foo is deprecated. It was supposed to
be gone from the tree as a user-setting a long time ago...
2014-04-25 19:25:37 +00:00
Warner Losh
d6d1ee211a Add option WITHOUT_VI to not build/install vi. vi was the largest
binary without a knob to turn it off.
2014-04-24 23:17:40 +00:00
Warner Losh
f6092768b1 Use MK_OPENLDAP in preference to WITH_OPENLDAP and make it a default
NO option to match the opt-in nature of the historical nature of this
option.
2014-04-24 23:17:31 +00:00
Warner Losh
04943ab399 Retire compatibility YES_HESIOD. If you haven't upgraded in the last
decade to WITH_HESIOD, your are in for a rude surprise...
2014-04-18 17:04:34 +00:00
Warner Losh
5144b2be23 Move the generic part of bsd.opts.mk into bsd.mkopts.mk to allow for
the WITH/WITHOUT_FOO -> MK_FOO={yes,no} stuff to be used elsewhere.
2014-04-18 17:04:26 +00:00
Warner Losh
4f9fa02d54 Allow setting of MK_FOO by anybody. There are too many edge cases that
we're preventing now with this policy. However, these edge cases
should be rare and all that set MK_FOO directly.
WITH_FOO and WITHOUT_FOO both being defined now result in the
non-default behavior happening silently. Users needing determinism
here fall into the edge case exception for MK_FOO setting.
2014-04-18 17:04:18 +00:00
Warner Losh
6a542ebed9 Copy bsd.own.mk to bsd.opts.mk and separate out the option setting
code from the rest. Include bsd.opts.mk in bsd.own.mk to preserve
current behavior. Future revisions will replace the inclusion of
bsd.own.mk elsewhere with bsd.opts.mk or a more appropriate new
file that's still being finalized.
2014-04-18 17:04:09 +00:00