Commit Graph

317 Commits

Author SHA1 Message Date
Enji Cooper
9a1f7ce50c Revert change (r322952) that was not yet destined for ^/head
This unbreaks the build.

This happened because of a botched "svn switch".

Reported by:	cem
2017-08-27 17:08:08 +00:00
Enji Cooper
b5197de15e Try and rebase the bsd.lib.mk changes after ^/head@r322824 was merged in
Unfortunately the snippet's now broken -- need to get the matching expressions to
work properly.
2017-08-27 16:46:51 +00:00
John Baldwin
de6feefdb7 Improve the coverage of debug symbols for MK_DEBUG_FILES.
- Include debug symbols in static libraries.  This permits binaries
  to include debug symbols for functions obtained from static libraries.
- Permit the C/C++ compiler flags added for MK_DEBUG_FILES to be
  overridden by setting DEBUG_FILES_CFLAGS.  Use this to limit the debug
  information for llvm libraries and binaries.

Reviewed by:	emaste
Sponsored by:	DARPA / AFRL
Differential Revision:	https://reviews.freebsd.org/D12025
2017-08-23 23:30:25 +00:00
Bryan Drewery
04594feee5 Use -S for library installations except for -DNO_ROOT builds.
Also disable this if NO_SAFE_LIBINSTALL is defined.

There is little harm in always using -S and it fixes several issues:
- A race during 'make libraries' where, for example, libgcc_s is being
  installed while another library is trying to link against it.  This is
  possible because libgcc_s is connected in both _prereq_libs and
  _startup_libs.  The first build (_prereq_libs) sets MK_PROFILE=no
  while the 2nd pass (_startup_libs) enables MK_PROFILE.  Thus the
  libgcc_s library *is* present in WORLDTMP for other libraries to
  link to, so serializing further items in _startup_libs is not
  required.  Just ensuring that libgcc_s is installed atomically (via
  rename(2)) is enough. [1]
- Installation to a running system where some library that cannot be
  detected, copied and used from the temporary INSTALLTMP with LD_LIBRARY_PATH
  that the build itself uses for installation.  Such an example is having the
  install an NSS module for user lookups that install(1) uses while
  concurrently installing the module in another process.  This is not
  a problem for the FreeBSD base build but can be for downstream
  vendors.  While this is a very specific case, installation to a
  running system with non-atomic library installation is prone to many
  problems.  A further step still is to install in proper dependency
  ordering.

Reported by:	dhw many times [1]
Sponsored by:	Dell EMC Isilon
MFC after:	2 weeks
2017-08-16 05:02:31 +00:00
Enji Cooper
38f8fddf05 Add limited sandbox capability to "make check"
== Rationale ==

r295380 introduced "make check" and consolidated means for running
test code in an attempt to simplify running tests. One could either
install files/libraries/programs and run "make check", or run "make check"
with an explicit CHECKDIR, e.g., `make check CHECKDIR=$(make -V.OBJDIR)``.

One criticism that was received is that "make check" should be run with
the intent of making dev->test->commit easier, which means that the target
audience's workflow should be developers. One developer pattern available
in other opensource projects is to run test code from a developer sandbox,
instead of installing to a system.

== Method ==

This approach is slightly different from the standard approach, in the sense
that it builds and installs into a deterministic directory under .OBJDIR (as I call it,
the "sandbox"), then runs "make check" against that. In the event the test
run is successful, the deterministic directory is removed to save space.

== Approach ==

bsd.lib.mk, bsd.prog.mk:

To support this functionality, a new variable `HAS_TESTS` is being added.

HAS_TESTS enables appropriate behavior with bsd.lib.mk and bsd.prog.mk, as
follows:
- Add "make check" as an available target from the directory.
- Pass down appropriate variables via ${TESTS_ENV}, i.e.,
  ${TESTS_LD_LIBRARY_PATH} and ${TESTS_PATH}.

One should add "HAS_TESTS" to directories containing tests in them, e.g. from
bin/sh/Makefile,

  HAS_TESTS=
  SUBDIR.${MK_TESTS}+= tests

HAS_TESTS doesn't automatically add the tests subdirectory for flexibility
reasons.

bsd.opts.mk, src.opts.mk:
- The knob ${MK_MAKE_CHECK_USE_SANDBOX} has been added, both to explicitly
  direct (internally) when to set a deterministic ${DESTDIR} and to also allow
  users to disable this behavior globally, i.e., via src.conf.
- MK_TESTS has been promoted from src.opts.mk to bsd.opts.mk to leverage
  syntactic sugar for having MK_TESTS be a dependency for
  MK_MAKE_CHECK_USE_SANDBOX, but to also ensure that src.opts.mk isn't required
  to use suite.test.mk (which is a dependency of bsd.test.mk).

suite.test.mk:
- beforecheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified
  from a no-op to:
-- Build.
-- Run "make hierarchy" on the sandbox dir.
-- Install the tests/files to the sandbox dir.
- aftercheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified
  from a no-op to:
-- Remove the sandbox dir.

Again, because the dependency order set in bsd.test.mk is
beforecheck -> check -> aftercheck, "make check" will not be run unless
"beforecheck" completes successfully, and "aftercheck" will not be run unless
"beforecheck" and "check" complete successfully.

== Caveats ==

- This target must either be run with MK_INSTALL_AS_USER or as root. Otherwise
  it will fail when running "make install" as the default user/group for many
  makefiles when calling INSTALL is root/wheel.
- This target must be run from a suitable top-level directory. For example,
  running tests from `tests/sys/fs/tmpfs` won't work, but `tests/sys/fs` will,
  because `tests/sys/fs/tmpfs` relies on files installed by `tests/sys/fs`.
- Running MK_INSTALL_AS_USER may introduce determinism issues. However, using
  it could identify deficiences in tests in terms of needing to be run as
  root, which are not properly articulated in the test requirements.
- The doesn't negate the need for running "make installworld" and
  "make checkworld", etc. Again, this just is intended to simplify the
  dev->test->commit workflow.

== Cleanup done ==
- CHECKDIR is removed; one can use "MK_MAKE_CHECK_USE_SANDBOX=no" to enable
  "legacy" (r295380) behavior.

MFC after:	2 months
Relnotes:	yes (CHECKDIR removed; "make check" behavior changed)
Requested by:	jhb
Reviewed by:	arch (silence), testing (silence)
Differential Revision:	D11905
2017-08-14 19:03:05 +00:00
Bryan Drewery
1deef6e403 Always set CLEANFILES, even if not building.
This will ensure that 'make clean' does the right thing.

MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2016-11-13 00:10:56 +00:00
Jonathan Anderson
e4195e2e12 Add rules to build LLVM IR binaries and libraries.
Running `make libfoo.ll` or `make libfoo.bc` within a library directory
will now give us an LLVM IR version of the library, and `make foo.full.ll`
or `make foo.full.bc` will give us an IR version of a binary.

As part of this change, we add an LLVM_LINK variable to sys.mk that can be
specified/overridden using an external toolchain.

Reviewed by:	bdrewery, brooks
Approved by:	rwatson (mentor)
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D8388
2016-11-01 21:27:42 +00:00
Marcel Moolenaar
e7c08366e7 Add LORDER, TSORT and TSORTFLAGS variables and replace the
hardcoded utility names and tsort flags.
2016-10-18 01:42:42 +00:00
Jonathan Anderson
808b18e4d6 Extract suffix rules into bsd.suffixes[-posix].mk.
Refactor make suffix rules into separate files (one for POSIX and one not),
and rationalise the rules so that bsd.lib.mk can contain only those rules
that are library-specific (.c.po and .c.pico).

This can be accomplished by adding ${STATIC_CFLAGS} to the .c.o rule
unconditionally. STATIC_CFLAGS are only defined for use by sys.mk rules in
lib/libpam/Makefile.inc (see r227797), so it should be safe to include
them unconditionally in sys.mk's .c.o rule (tested by make universe and a
ports exp-run).

Reviewed by:	bdrewery, sjg
Approved by:	rwatson (mentor)
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D6805
2016-10-12 00:42:46 +00:00
Marcel Moolenaar
2b4da8aa20 When MAKEOBJDIRPREFIX points to a case-insensitive file system, the
build can break when different source files create the same target
files (case-insensitivity speaking).  This is the case for object
files compiled with -fpic and shared libraries. The former uses
an extension of ".So", and the latter an extension ".so".  Rename
shared object files from *.So to *.pico to match what NetBSD does.

See also r305855

MFC after:	1 month
Sponsored by:	Bracket Computing
Differential Revision:	https://reviews.freebsd.org/D7906
2016-09-24 15:11:27 +00:00
Bryan Drewery
919154d9c1 Don't truncate OBJS_DEPEND_GUESS.target from Makefile
This is important to allow a Makefile to override OBJS_DEPEND_GUESS for
handling in META_MODE when its depend files are missing.

Approved by:	re (implicit)
Sponsored by:	EMC / Isilon Storage Division
2016-06-15 23:57:32 +00:00
Bryan Drewery
447bb5a3bd WITH_META_MODE+WITH_DEBUG_FILES: Fix library symlinks causing bogus rebuilds.
A simplified example of the library targets with WITH_DEBUG_FILES is:

  libgeom.so.5: libgeom.so.5.full
     cp libgeom.so.5.full libgeom.so.5

  libgeom.so.5.full:
     ln -s libgeom.so.5 libgeom.so
     cc -o libgeom.so.5.full *.o

Before, or without, WITH_DEBUG_FILES it is:

  libgeom.so.5:
     ln -s libgeom.so.5 libgeom.so
     cc -o libgeom.so.5 *.o

The problem is that bmake considers the link source for the libgeom.so
link in the libgeom.so.5.full target as being a dependency for
libgeom.so.5.full.  That resolves to libgeom.so.5.  Thus a cyclic
dependency is created.  The result of this is that if libgeom.so.5 is
created with a newer timestamp than libgeom.so.5.full, then
libgeom.so.5.full will be rebuilt on the next build.  This causes a
chain reaction of everything in the build relinking, or hitting the
problem itself.

Moving the link creation to the target that actually creates
libgeom.so.5 fixes the problem.  The simplest fix here is to just
duplicate the logic.

Submitted by:	sjg
Approved by:	re (implicit)
2016-06-14 16:19:44 +00:00
Bryan Drewery
5852ae2d99 Revert r301079.
This breaks cross-building with WITH_META_MODE since it will rebuild
'build-tools' during the 'everything' phase.

A more proper fix is coming to bmake to implicitly require .META unless
.NOMETA (and other restrictions) are in place.
2016-06-03 19:25:36 +00:00
Bryan Drewery
28849c57d6 WITH_META_MODE: Mitigate switching from without to with META_MODE.
Adding .META to targets-to-build will ensure that they will rebuild if there
is no .meta file.

Adding it to all SUFFIXES and objects ensures that at least objects will
rebuild if there is no .meta file.

This will be reverted if bmake's behavior changes to rebuild on missing .meta
files.

Sponsored by:	EMC / Isilon Storage Division
2016-05-31 21:22:19 +00:00
Ed Maste
ce9d1e2acd Deorbit ALLOW_SHARED_TEXTREL
We want to avoid .text relocations in shared objects. libcrypto was the
only consumer and it is now fixed (as of r299389). Remove the now-unused
support for turning off the linker warning.

Reviewed by:	kib
Differential Revision:	https://reviews.freebsd.org/D6323
2016-05-11 17:55:09 +00:00
Glen Barber
0edd2576c0 MFH
Sponsored by:	The FreeBSD Foundation
2016-04-16 02:32:12 +00:00
Bryan Drewery
21e2745698 Add SHLIB_CXX to allow building a C++ shared library without a static one.
Submitted by:	ngie
Sponsored by:	EMC / Isilon Storage Division
2016-04-15 18:43:54 +00:00
Glen Barber
d60840138f MFH
Sponsored by:	The FreeBSD Foundation
2016-04-04 23:55:32 +00:00
Bryan Drewery
497e80911e Remove the old depend (mkdep) code and make FAST_DEPEND the one true way.
Reviewed by:	emaste, hselasky (partial), brooks (brief)
Discussed on:	arch@
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D5742
2016-03-30 23:50:23 +00:00
Bryan Drewery
1b337a349d Build libcompat (lib32) with a --sysroot pointing into its stage directory.
This overrides the cross-compiler's default sysroot to use the WORLD32's
sysroot for building the lib32 libraries.  Previously the cross-compiler
would default the sysroot to the 64bit WORLDTMP and -B/-L/-isystem flags
were used to build using the lib32 files.  This leads to multiple issues
discussed later.  Some extra headers are now needed to be staged since the
64bit WORLDTMP is not referenced at all for headers.  The 64bit WORLDTMP
is still used via PATH for build tools.  Overriding the default
target/arch is retained in the CC/CXX overrides.

This allows reverting the LDSCRIPT rewriting in installworld from r296921 and
r235122, thus allowing read-only objdirs to work for installing again.

This removes the need for _LDSCRIPTROOT.

This allows progressing the change to always use --sysroot for the build
rather than only relying on the cross-compiler's default sysroot.  The
work for that is in D3970 and needed to resolve WITHOUT_CROSS_COMPILER
not using a --sysroot [1].

PR:		196193 [1]
Sponsored by:	EMC / Isilon Storage Division
2016-03-25 19:12:38 +00:00
Bryan Drewery
54bc9e303d Partially revert r266227 and stop stripping paths in ldscripts.
Specifically this fixes /usr/lib/libc.so stripping the paths to the
libraries.  The reason for this in r266227 was both likely because ld(1) did
not fully respect --sysroot until r291226 and because of the lib32
build.  The lib32 build does not use --sysroot into the /usr/lib32 path,
rather it only uses -L and -B into the /usr/lib32 path and --sysroot
into the normal (64bit) /usr/lib.  The _LDSCRIPTROOT was added with
the ldscript support in bsd.lib.mk so that it builds a 32-bit-sysrooted pathed
ldscript in the object directory and then installs a normal unprefixed
version in installworld.  This commit also fixes the rebuild during
install which was broken in r266227.  This commit would break DIRDEPS_BUILD
build of lib32 but it does not currently have a way to build it anyhow.

For example, before this change we had in /usr/lib/libc.so:
  GROUP ( libc.so.7 libc_nonshared.a libssp_nonshared.a )
Now it is restored to pre-r266227:
  GROUP ( /lib/libc.so.7 /usr/lib/libc_nonshared.a /usr/lib/libssp_nonshared.a )

The motivation for this is in testing of lld.
From emaste:
  lld does not have built-in search paths (e.g. /lib, /usr/lib) and relies on
  -L arguments passed by the caller.  As the linker is nearly always invoked
  from the clang driver this is fine other than the fact that /usr/lib/libc.so
  is an ldscript that refers to libc.so.7 which is in /lib, not /usr/lib.

PR:		207980
Reported by:	emaste
Submitted by:	emaste (based on)
Differential Revision:	https://reviews.freebsd.org/D5637
2016-03-15 22:27:29 +00:00
Glen Barber
7d536dc855 MFH
Sponsored by:	The FreeBSD Foundation
2016-03-10 21:16:01 +00:00
Bryan Drewery
c12b4204c6 Follow-up r296324: Fix STATICOBJS dependency guesses conditions.
Reported by:	antoine
Pointyhat to:	bdrewery
Sponsored by:	EMC / Isilon Storage Division
2016-03-08 16:12:55 +00:00
Glen Barber
b655ec9752 MFH
Sponsored by:	The FreeBSD Foundation
2016-03-06 04:13:17 +00:00
Bryan Drewery
ee05c0f1b8 Allow adding to SOLINKOPTS.
Sponsored by:	EMC / Isilon Storage Division
2016-03-04 22:37:00 +00:00
Bryan Drewery
a324b0f04c DEBUG_FILES: Create the DEBUGFILEDIR if it doesn't exist on install.
Currently the base.txz distribution does not get the BSD.debug.dist mtree
extracted into it.  So if you start from that and then try to build a 3rd-party
application outside of buildworld it will by-default try installing the
debug files into a missing directory if they are being installed into /usr/lib.

Check for the existence before forcing the directory to be created rather than
the older way of running a shell command with test -d || mkdir -p always.

Reported by:	HardenedBSD (https://github.com/HardenedBSD/secadm/issues/23)
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D5411
2016-03-03 18:08:58 +00:00
Glen Barber
42d27ee343 MFH
Sponsored by:	The FreeBSD Foundation
2016-03-02 23:53:08 +00:00
Bryan Drewery
88fe1f7ab3 Add guessed dependencies to OBJS after bsd.dep.mk in case of it adding to SRCS.
This was a regression in r295985.

bsd.dep.mk adds to SRCS for dtrace probes, yacc grammars and some
others.

The code that is moving is planned to be removed once FAST_DEPEND is
default (and the only option) though since FAST_DEPEND doesn't use this.

Pointyhat to:	bdrewery
Sponsored by:	EMC / Isilon Storage Division
2016-03-02 21:03:42 +00:00
Glen Barber
52259a98ad MFH
Sponsored by:	The FreeBSD Foundation
2016-03-02 16:14:46 +00:00
Bryan Drewery
aae63957af Import bsd.clang-analyze.mk based on NetBSD's version.
This allows 'make analyze' or 'make OBJ.clang-analyzer' to run the
Clang static analyzer and present results on stdout.

Obtained from:	NetBSD (CVS Rev. 1.3)
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D5449
2016-02-26 22:14:00 +00:00
Bryan Drewery
f1702f873d Remove _manpages and _maninstall targets as they are no longer needed.
r96164 added them to avoid recursing twice with _SUBDIR.  That issue was
fixed in bsd.subdir.mk in r291635 for all targets.

Sponsored by:	EMC / Isilon Storage Division
2016-02-26 22:13:31 +00:00
Bryan Drewery
d7beff082b FAST_DEPEND: Rework how guessed dependencies are handled.
Rather than depend on .depend not existing, check the actual
.depend.OBJ file that will be used for that object.  If it doesn't
exist then use the guessed dependencies.

FAST_DEPEND may never have a .depend file.  Not having one means all of the
previous logic would over-depend all object files on all headers which is not
what we wanted.  It also means that if a .depend is generated before a build
is done for _EXTRADEPEND (such as for PROG or LIB) then all of these
dependencies would not be used since the .depend wasn't generated from mkdep
and the real .depend.* files are not generated until the build.

Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:19:05 +00:00
Glen Barber
72c3aa02dc MFH
Sponsored by:	The FreeBSD Foundation
2016-02-18 00:37:58 +00:00
Bryan Drewery
1bd4272fb9 Consolidate common beforebuild logic.
Sponsored by:	EMC / Isilon Storage Division
2016-02-16 02:09:48 +00:00
Glen Barber
59d43d11fe MFH
Sponsored by:	The FreeBSD Foundation
2016-01-20 09:50:54 +00:00
Bryan Drewery
eb2ad8724b FAST_DEPEND: Fix improperly depending all .So objects on all headers.
This was a regression in r290629, which was revealed partly in r294360.
Once 'make depend' has ran it will generate all headers already.  Thus
even with FAST_DEPEND lacking proper dependencies before building, it
will not have any missing headers.  Once objects are compiled the depend
files will be generated with proper dependencies.

Sponsored by:	EMC / Isilon Storage Division
2016-01-19 23:28:18 +00:00
Glen Barber
b626f5a73a MFH r289384-r293170
Sponsored by:	The FreeBSD Foundation
2016-01-04 19:19:48 +00:00
Colin Percival
f051a5b324 Make libxnet.so a symlink to libc.so. This makes -lxnet a no-op, as
POSIX requires for the c99 compiler.

(In fact, our c99(1) already ignores -lxnet; but our make(1) doesn't set
${CC} correctly, and our cc(1) treats xnet like any other library.)

Reviewed by:	kib
2015-12-25 11:29:18 +00:00
Bryan Drewery
10f696a52d FAST_DEPEND: Only pass -MF if we care about the object being compiled.
This will save time generating dependency files that we didn't expect
due to cases where SRCS!=OBJS or for building custom targetted objects
in Makefiles that do not end up in the DEPENDOBJS list.

This uses a bmake trick to modify CFLAGS based on ${.TARGET}.  A
.PARSEDIR check is done for the sake of MFC safety.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-12-07 16:08:22 +00:00
Bryan Drewery
f35c8b65a6 FAST_DEPEND: Similar to r290629, do always depend on headers if 'make depend'
has not ran yet.

This fixes building objects directly, or skipping 'make depend', not generating
required headers first.  This case did work without FAST_DEPEND so there's no
reason it should not work here as well.

An example of this can be seen building in gnu/usr.bin/binutils/libbfd
without running 'make depend' first to generate config.h.

Sponsored by:	EMC / Isilon Storage Division
MFC after:	3 weeks
X-MFC-With:	r290433
2015-11-19 00:45:36 +00:00
Bryan Drewery
b03ca5c3ec FAST_DEPEND: Fix building of wrong source files in some cases.
Similar to the original reason for these dependency hints to be added,
in r124637, the missing-dependency file case can lead to building of the
wrong source.

A clear example of this is in gnu/lib/libstdc++ where the .PATH contains
both contrib/gcc and contrib/libstdc++/src.

  contrib/gcc has a debug.c.
  contrib/libstdc++/src has a debug.cc.

  When building for the objects of debug.o, debug.So, and debug.po, it is
  ambiguous for which src file to use due to the suffix transformation
  rules, even though the proper one is listed first in .PATH.

  This was normally avoided due to these dependency hints for the initial
  build, and then mkdep would add an explicit 'debug.o: debug.cc'
  dependency into the .depend file.  WITH_FAST_DEPEND does not generate
  the .depend file with these, but puts them into .depend.debug.o instead.

Rather than extending the exists() check to each object's .depend.*
file, just enable the hint when when using WITH_FAST_DEPEND.  It fixes
the problem and seems to be safe enough to use since it is mapping SRCS
back to OBJS, rather than letting make make assumptions from OBJS to
SRCS.

A similar check mapping objects to headers is present in some mk files
but was not extended here for FAST_DEPEND since it has not yet been
found to be a problem.

X-MFC-With:	r290433
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-11-09 23:37:04 +00:00
Bryan Drewery
9160419c7a Add built-in ccache build support via WITH_CCACHE_BUILD option.
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build.  This is explained in
more detail upstream [1] [2].  It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times.  When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'.  Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports.  This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead.  ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss.  For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct.  Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it.  Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings.  GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual.  Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization.  There is also a memcached backend available [5].  Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout.  A
possible working layout is as follows:
  Source: /some/prefix/src1
  Source: /some/prefix/src2
  Source: /some/prefix/src3
  Objdir: /some/prefix/obj
  Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative.  Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files and ccache objects.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
  x buildworld-before
  + buildworld-ccache-empty
  * buildworld-ccache-full
  % buildworld-ccache-full-fastdep
  # buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |%            *                               #                                +|
  |%            *                               #                                +|
  |%            *                               #             xxx                +|
  |                                                           |A                  |
  |                                                                              A|
  |             A                                                                 |
  |A                                                                              |
  |                                             A                                 |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3          4519       4525.04       4520.73       4521.59     3.1104823
  Difference at 95.0% confidence
          758.027 +/- 43.4565
          20.1412% +/- 1.15466%
          (Student's t, pooled s = 19.1726)
  *   3       1823.08        1827.2       1825.62        1825.3     2.0785572
  Difference at 95.0% confidence
          -1938.26 +/- 43.298
          -51.5007% +/- 1.15045%
          (Student's t, pooled s = 19.1026)
  %   3       1266.96       1279.37       1270.47     1272.2667     6.3971113
  Difference at 95.0% confidence
          -2491.3 +/- 44.3704
          -66.1952% +/- 1.17895%
          (Student's t, pooled s = 19.5758)
  #   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-ccache-empty
  * buildkernel-ccache-empty-fastdep
  % buildkernel-ccache-full
  # buildkernel-ccache-full-fastdep
  @ buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |#                        @   %                  *                              |
  |#                        @   %                  *     x                      + |
  |#                        @   %                  *     xx                     ++|
  |                                                      MA                       |
  |                                                                             MA|
  |                                                A                              |
  |                             A                                                 |
  |A                                                                              |
  |                         A                                                     |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        727.97        731.91        728.06     729.31333     2.2492295
  Difference at 95.0% confidence
          156.88 +/- 4.17129
          27.4058% +/- 0.728695%
          (Student's t, pooled s = 1.84034)
  *   3         527.1        528.29        528.08     527.82333    0.63516402
  Difference at 95.0% confidence
          -44.61 +/- 2.33254
          -7.79305% +/- 0.407478%
          (Student's t, pooled s = 1.02909)
  %   3         400.4        401.05        400.62        400.69     0.3306055
  Difference at 95.0% confidence
          -171.743 +/- 2.16453
          -30.0023% +/- 0.378128%
          (Student's t, pooled s = 0.954969)
  #   3        201.94        203.34        202.28        202.52    0.73020545
  Difference at 95.0% confidence
          -369.913 +/- 2.40293
          -64.6212% +/- 0.419774%
          (Student's t, pooled s = 1.06015)
  @   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR:		182944 [4]
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
Relnotes:	yes
2015-11-08 00:50:18 +00:00
Bryan Drewery
cf1eeb33be Add a FAST_DEPEND option, off by default, which speeds up the build significantly.
This speeds up buildworld by 16% on my system and buildkernel by 35%.

Rather than calling mkdep(1), which is just a wrapper around 'cc -E',
use the modern -MD -MT -MF flags to gather and generate dependencies during
compilation.  This flag was introduced in GCC "a long time ago", in GCC 3.0,
and is also supported by Clang.  (It appears that ICC also supports this but I
do not have access to test it).  This avoids running the preprocessor *twice*
for every build, in both 'make depend' and 'make all'.  This is especially
noticeable when using ccache since it does not cache preprocessor results from
mkdep(1) / 'cc -E', but still speeds up compilation with the -MD flags.

For 'make depend' a tree-walk is still done to ensure that all DPSRCS
are generated when expected, and that beforedepend/afterdepend and
_EXTRADEPEND are all still respected.  In time this may change but for now
I've been conservative.  The time for a tree-walk with -j combined with
SUBDIR_PARALLEL is not significant.  For example, it takes about 9 seconds
with -j15 to walk all of src/ for 'make depend' now on my system.

A .depend file is still generated with the various rules that apply to
the final target, or custom rules.  Otherwise there are now
per-built-object-file .depend files, such as .depend.filename.o.  These
are included directly by make rather than populating .depend with a loop
and .depend lines, which only added overhead to the now almost-NOP 'make
depend' phase.

Before this I experimented with having mkdep(1) called in parallel per-file.
While this improved the kernel and lib/libc 'make depend' phase, it resulted
in slower build times overall.

The -M flags are removed from CFLAGS when linking since they have no effect.

Enabling this by default, for src or out-of-src, can be done once more testing
has been done, such as a ports exp-run, and with more compilers.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_FAST_DEPEND=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

buildworld:
  x buildworld-before
  + buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |+                                                                              |
  |+                                                                              |
  |+                                                                       xx    x|
  |                                                                       |_MA___||
  |A                                                                              |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |+                                                                            x |
  |++                                                                           xx|
  |                                                                             A||
  |A|                                                                             |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

Sponsored by:	EMC / Isilon Storage Division
MFC after:	3 weeks
Relnotes:	yes
2015-11-06 04:45:29 +00:00
Bryan Drewery
29f297420d Similar to r289355, /usr/tests is within the base system so put the symbols
into /usr/lib/debug.

This covers some missing files:
  /usr/tests/libexec/rtld-elf/.debug
  /usr/tests/libexec/rtld-elf/.debug/libpythagoras.so.0.debug
  /usr/tests/lib/libc/tls/.debug
  /usr/tests/lib/libc/tls/.debug/libh_tls_dynamic.so.1.debug
  /usr/tests/lib/libc/tls/.debug/h_tls_dlopen.so.debug
  /usr/tests/lib/libthr/dlopen/.debug
  /usr/tests/lib/libthr/dlopen/.debug/h_pthread_dlopen.so.1.debug
  /usr/tests/lib/libxo/.debug
  /usr/tests/lib/libxo/.debug/libenc_test.so.debug

Sponsored by:	EMC / Isilon Storage Division
2015-10-16 05:06:43 +00:00
Bryan Drewery
61c20fc782 Make installing to a non-existent directory an error.
Before this, if a file was installed to DESTDIR/some/dir and that directory
was missing due to not having ran 'make distrib-dirs' yet, the file would
be installed as 'some/dir'.  For something like bsd.incs.mk with INCLUDEDIR
being a sub-directory of /usr/include, this could result in all of the headers
being installed to a file rather than getting a directory of them.

Now it will error that the file/directory does not exist rather than hide
the issue.

Another option being discussed is to implement GNU's install -D flag which
would auto create any missing directories.

This is a mitigation of the problem.  The proper order to the build is to
run 'make distrib-dirs' first, but that can be forgotten if building from
a sub-directory after updating the source code to the latest revision.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-10-15 22:49:56 +00:00
Baptiste Daroussin
a2fef6b9c1 Fix bad merge 2015-10-11 15:39:37 +00:00
Baptiste Daroussin
bfa9859576 Fix typo 2015-10-11 15:11:53 +00:00
Baptiste Daroussin
9be27fdc00 Merge from head 2015-10-09 22:45:54 +00:00
Baptiste Daroussin
9b4f4918ca Add a new bsd.confs.mk similar to bsd.files.mk or bsd.incs.mk
It defines a CONFS variable for all files supposed to be installed as a
configuration file and handle as such
2015-10-09 21:57:42 +00:00
Brad Davis
09923f9c11 Fix a typo that was causing lines like the following in the METALOG:
./.else type=link mode=0755 link=../../stage/lib/libm.so.5 tags=package=runtime,development
./.else type=link mode=0755 link=../../stage/lib/libgeom.so.5 tags=package=runtime,development

Which was causing packaging to fail.

Approved by:	bapt
2015-10-06 14:56:37 +00:00