kgzldr.o is installed into /usr/lib but using bsd.prog.mk. Add
/usr/lib to the base system directory list so that debug files are
installed into /usr/lib/debug/usr/lib, not /usr/lib/.debug .
Approved by: re (gjb)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
The DPADD data in .depend will be redundant with what is in the .meta file.
Also extend NO_EXTRADEPEND support to bsd.prog.mk.
Approved by: re (blanket, META_MODE)
Sponsored by: EMC / Isilon Storage Division
This breaks cross-building with WITH_META_MODE since it will rebuild
'build-tools' during the 'everything' phase.
A more proper fix is coming to bmake to implicitly require .META unless
.NOMETA (and other restrictions) are in place.
Adding .META to targets-to-build will ensure that they will rebuild if there
is no .meta file.
Adding it to all SUFFIXES and objects ensures that at least objects will
rebuild if there is no .meta file.
This will be reverted if bmake's behavior changes to rebuild on missing .meta
files.
Sponsored by: EMC / Isilon Storage Division
Currently the base.txz distribution does not get the BSD.debug.dist mtree
extracted into it. So if you start from that and then try to build a 3rd-party
application outside of buildworld it will by-default try installing the
debug files into a missing directory if they are being installed into /usr/lib.
Check for the existence before forcing the directory to be created rather than
the older way of running a shell command with test -d || mkdir -p always.
Reported by: HardenedBSD (https://github.com/HardenedBSD/secadm/issues/23)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5411
This was a regression in r295985.
bsd.dep.mk adds to SRCS for dtrace probes, yacc grammars and some
others.
The code that is moving is planned to be removed once FAST_DEPEND is
default (and the only option) though since FAST_DEPEND doesn't use this.
Pointyhat to: bdrewery
Sponsored by: EMC / Isilon Storage Division
This allows 'make analyze' or 'make OBJ.clang-analyzer' to run the
Clang static analyzer and present results on stdout.
Obtained from: NetBSD (CVS Rev. 1.3)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5449
r96164 added them to avoid recursing twice with _SUBDIR. That issue was
fixed in bsd.subdir.mk in r291635 for all targets.
Sponsored by: EMC / Isilon Storage Division
Rather than depend on .depend not existing, check the actual
.depend.OBJ file that will be used for that object. If it doesn't
exist then use the guessed dependencies.
FAST_DEPEND may never have a .depend file. Not having one means all of the
previous logic would over-depend all object files on all headers which is not
what we wanted. It also means that if a .depend is generated before a build
is done for _EXTRADEPEND (such as for PROG or LIB) then all of these
dependencies would not be used since the .depend wasn't generated from mkdep
and the real .depend.* files are not generated until the build.
Sponsored by: EMC / Isilon Storage Division
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build. This is explained in
more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times. When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'. Stats are provided at the end of this message.
This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.
The option name was picked to match the existing option in ports. This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.
Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead. ccache does nothing special in these modes, although there is
no harm in calling it for them.
CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss. For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct. Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.
The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it. Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings. GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.
Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual. Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization. There is also a memcached backend available [5]. Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout. A
possible working layout is as follows:
Source: /some/prefix/src1
Source: /some/prefix/src2
Source: /some/prefix/src3
Objdir: /some/prefix/obj
Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative. Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.
distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.
The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).
Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.
The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files and ccache objects.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16
The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.
buildworld:
x buildworld-before
+ buildworld-ccache-empty
* buildworld-ccache-full
% buildworld-ccache-full-fastdep
# buildworld-fastdep
+-------------------------------------------------------------------------------+
|% * # +|
|% * # +|
|% * # xxx +|
| |A |
| A|
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 4519 4525.04 4520.73 4521.59 3.1104823
Difference at 95.0% confidence
758.027 +/- 43.4565
20.1412% +/- 1.15466%
(Student's t, pooled s = 19.1726)
* 3 1823.08 1827.2 1825.62 1825.3 2.0785572
Difference at 95.0% confidence
-1938.26 +/- 43.298
-51.5007% +/- 1.15045%
(Student's t, pooled s = 19.1026)
% 3 1266.96 1279.37 1270.47 1272.2667 6.3971113
Difference at 95.0% confidence
-2491.3 +/- 44.3704
-66.1952% +/- 1.17895%
(Student's t, pooled s = 19.5758)
# 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)
buildkernel:
x buildkernel-before
+ buildkernel-ccache-empty
* buildkernel-ccache-empty-fastdep
% buildkernel-ccache-full
# buildkernel-ccache-full-fastdep
@ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|# @ % * |
|# @ % * x + |
|# @ % * xx ++|
| MA |
| MA|
| A |
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 727.97 731.91 728.06 729.31333 2.2492295
Difference at 95.0% confidence
156.88 +/- 4.17129
27.4058% +/- 0.728695%
(Student's t, pooled s = 1.84034)
* 3 527.1 528.29 528.08 527.82333 0.63516402
Difference at 95.0% confidence
-44.61 +/- 2.33254
-7.79305% +/- 0.407478%
(Student's t, pooled s = 1.02909)
% 3 400.4 401.05 400.62 400.69 0.3306055
Difference at 95.0% confidence
-171.743 +/- 2.16453
-30.0023% +/- 0.378128%
(Student's t, pooled s = 0.954969)
# 3 201.94 203.34 202.28 202.52 0.73020545
Difference at 95.0% confidence
-369.913 +/- 2.40293
-64.6212% +/- 0.419774%
(Student's t, pooled s = 1.06015)
@ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)
[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30
PR: 182944 [4]
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
Relnotes: yes
This speeds up buildworld by 16% on my system and buildkernel by 35%.
Rather than calling mkdep(1), which is just a wrapper around 'cc -E',
use the modern -MD -MT -MF flags to gather and generate dependencies during
compilation. This flag was introduced in GCC "a long time ago", in GCC 3.0,
and is also supported by Clang. (It appears that ICC also supports this but I
do not have access to test it). This avoids running the preprocessor *twice*
for every build, in both 'make depend' and 'make all'. This is especially
noticeable when using ccache since it does not cache preprocessor results from
mkdep(1) / 'cc -E', but still speeds up compilation with the -MD flags.
For 'make depend' a tree-walk is still done to ensure that all DPSRCS
are generated when expected, and that beforedepend/afterdepend and
_EXTRADEPEND are all still respected. In time this may change but for now
I've been conservative. The time for a tree-walk with -j combined with
SUBDIR_PARALLEL is not significant. For example, it takes about 9 seconds
with -j15 to walk all of src/ for 'make depend' now on my system.
A .depend file is still generated with the various rules that apply to
the final target, or custom rules. Otherwise there are now
per-built-object-file .depend files, such as .depend.filename.o. These
are included directly by make rather than populating .depend with a loop
and .depend lines, which only added overhead to the now almost-NOP 'make
depend' phase.
Before this I experimented with having mkdep(1) called in parallel per-file.
While this improved the kernel and lib/libc 'make depend' phase, it resulted
in slower build times overall.
The -M flags are removed from CFLAGS when linking since they have no effect.
Enabling this by default, for src or out-of-src, can be done once more testing
has been done, such as a ports exp-run, and with more compilers.
The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_FAST_DEPEND=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16
buildworld:
x buildworld-before
+ buildworld-fastdep
+-------------------------------------------------------------------------------+
|+ |
|+ |
|+ xx x|
| |_MA___||
|A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)
buildkernel:
x buildkernel-before
+ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|+ x |
|++ xx|
| A||
|A| |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)
Sponsored by: EMC / Isilon Storage Division
MFC after: 3 weeks
Relnotes: yes
Before this, if a file was installed to DESTDIR/some/dir and that directory
was missing due to not having ran 'make distrib-dirs' yet, the file would
be installed as 'some/dir'. For something like bsd.incs.mk with INCLUDEDIR
being a sub-directory of /usr/include, this could result in all of the headers
being installed to a file rather than getting a directory of them.
Now it will error that the file/directory does not exist rather than hide
the issue.
Another option being discussed is to implement GNU's install -D flag which
would auto create any missing directories.
This is a mitigation of the problem. The proper order to the build is to
run 'make distrib-dirs' first, but that can be forgotten if building from
a sub-directory after updating the source code to the latest revision.
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
This mostly fixes an interaction with bsd.test.mk with PROGS and SCRIPTS.
This was most notable with 'make clean' and 'make install', which r281055
and r272055 attempted to address but were inadequate.
It also addresses similar issues in bsd.progs.mk when not using bsd.test.mk.
This also fixes cases of NOT running commands in the parent when using
bsd.progs.mk:
- 'make clean' was not run for the main process for Makefiles which had both
FILES and SUBDIR but no PROGS or SCRIPTS. This usually was just a
leftover Kyuafile.auto. One such example is usr.bin/bmake/tests/sysmk/t1/2.
- 'make obj' was not running in the current directory with bsd.test.mk due
to early inclusion of bsd.subdir.mk. This was not really a problem due to
the SUBDIRS using 'mkdir -p' for their objdirs.
There were subtle bugs causing this wrong behavior:
1. bsd.progs.mk needs to set SCRIPTS to empty when recursing to avoid
the sub-makes from installing, cleaning or building the SCRIPTS;
only the parent make should be doing this. r281055 effectively did
the same but wasn't enough.
2. CLEANFILES may contain (especially from *.test.mk) files which only
the parent should clean, such as from FILES and SCRIPTS. To resolve
sub-makes also cleaning these, reset CLEANFILES and CLEANDIRS in the
children before including bsd.prog.mk. A tempting alternative would be
to only handle CLEANFILES in the parent but then the child bsd.prog.mk
CLEANFILES of per-PROGS wouldn't be setup.
3. bsd.subdir.mk was included too soon in bsd.test.mk. It needs to be
included after bsd.prog.mk as the SCRIPTS logic is short-circuitted if
'install:' is already defined (which bsd.subdir.mk does). There is
actually no need to include bsd.subdir.mk from bsd.test.mk as bsd.prog.mk
and bsd.obj.mk will do so in the proper order. The description in r257095
covers this for FILES and was fixed differently, though changing the
handling of target(install) in bsd.prog.mk may make sense after more
research.
4. bsd.progs.mk had extra logic to handle recursing SCRIPTS if PROGS was
empty, which isn't its business to be doing. SCRIPTS is handled fine
by bsd.prog.mk. This mostly reverts and reworks the fix in r259209 and
partially reverts r272055.
5. bsd.progs.mk has no need to depend 'all:' on SCRIPTS and FILES. These
are handled by bsd.prog.mk/bsd.files.mk fine. This also partially reverts
r272055.
6. bsd.progs.mk was not drop-in safe for bsd.prog.mk. Move the PROGS
check from r273186 to allow it to be used safely.
Specific tested cases:
SCRIPTS:no PROGS:no FILES:yes SUBDIR:yes
usr.bin/bmake/tests/sysmk/t1/2
SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no
usr.bin/bmake/tests/sysmk/t1/2/1
SCRIPTS:yes PROGS:yes FILES:yes SUBDIR:yes
lib/libthr/tests
SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no
usr.bin/yacc/tests
libexec/atf/atf-sh/tests
A full buildworld/installworld/clean comparison with mtree was also done.
The only relevant difference was the new fixed behavior of removing
Kyuafile.auto from the objdir in 'clean'.
Converting SCRIPTS to be a special case FILES group will make this less
fragile and is being explored.
One known remaining issue is 'cleandepend' removing the tags files for
every recursive call.
Note that the 'make clean' command runs for the CURDIR last, which can make
it appear to run multiple times when cleaning in tests/, but each command is
for a SUBDIR returning up the chain. This is purely bsd.subdir.mk behavior.
PR: 191055
PR: 191955
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
Now when a lib is marked as PRIVATELIB it is renamed into libprivate$foo instead
of being installed in /usr/lib/private and playing with rpath.
Also allow to install headers for PRIVATELIBS in that case the headers will be
installed in /usr/include/private/$foo
Keep the headers under a private namespace to prevent third party build system
to easily find them to ensure they are only used on purpose.
This allows for non base applications to statically link against a library in
base which is linked to a privatelib
Treating PRIVATELIBS as regular libraries allows to push them into our current
compatX packages if needed.
While here finish promotion of libevent as PRIVATELIB
Install header for bsdstat and libucl
Differential Revision: https://reviews.freebsd.org/D2365
Reviewed by: brooks, des
Discussed with: imp
SHLIB_NAME_FULL so that the full binary is relinked when a dependency
changes. Right now the existing full binary is left as-is and only
the objcopy to remove debug symbols is run.
Differential Revision: https://reviews.freebsd.org/D1834
Reviewed by: emaste
MFC after: 3 days
update paths; and include everything in the "base" distribution.
The "games" distribution being optional made sense when there were more
games and we had small disks; but the "games-like" games were moved into
the ports tree a dozen years ago and the remaining "utility-like" games
occupy less than 0.001% of my laptop's small hard drive. Meanwhile every
new user is confronted by the question "do you want games installed" when
they they try to install FreeBSD.
The next steps will be:
2. Removing punch card (bcd, ppt), phase-of-moon (pom), clock (grdc), and
caesar cipher (caesar, rot13) utilities. I intend to keep fortune, factor,
morse, number, primes, and random, since there is evidence that those are
still being used.
3. Merging src/games into src/usr.bin.
This change will not be MFCed.
Reviewed by: jmg
Discussed at: EuroBSDCon
Approved by: gjb (release-affecting changes)
Otherwise there's nothing for ctfconvert to do, and it ends up emitting an
error for each object file. Also remove some redundant checks from
bsd.prog.mk and bsd.lib.mk.
Differential Revision: https://reviews.freebsd.org/D1111
Reviewed by: imp
This allows WITH_DEBUG_FILES to produce standalone debug for the ELF
runtime linker.
We previously disabled standalone debug files for bsd.prog.mk consumers
that included a non-default ${PROG} target, but this is not required.
Consumers that do not support standalone debug are still handled by
disabling it for statically linked binaries, and for those that specify
a non-default binary format.
Sponsored by: DARPA, AFRL
1. 50+% of NO_PIE use is fixed by adding -fPIC to INTERNALLIB and other
build-only utility libraries.
2. Another 40% is fixed by generating _pic.a variants of various libraries.
3. Some of the NO_PIE use is a bit absurd as it is disabling PIE (and ASLR)
where it never would work anyhow, such as csu or loader. This suggests
there may be better ways of adding support to the tree. Many of these
cases can be fixed such that -fPIE will work but there is really no
reason to have it in those cases.
4. Some of the uses are working around hacks done to some Makefiles that are
really building libraries but have been using bsd.prog.mk because the code
is cleaner. Had they been using bsd.lib.mk then NO_PIE would not have
been needed.
We likely do want to enable PIE by default (opt-out) for non-tree consumers
(such as ports). For in-tree though we probably want to only enable PIE
(opt-in) for common attack targets such as remote service daemons and setuid
utilities. This is also a great performance compromise since ASLR is expected
to reduce performance. As such it does not make sense to enable it in all
utilities such as ls(1) that have little benefit to having it enabled.
Reported by: kib
the build starts.
This adds a new variable DHDRS that contains a list of all DTrace
header files. Then, we use the beforedepend hook to make sure the
heaeder files are built.
Introduce a beforebuild dependency (from projects/bmake) based on
feedback from Simon J. Gerraty. This lets us generate the header
files without running make depend.
Reviewed by: sjg, imp
MFC after: 3 days
This is currently an opt-in build flag. Once ASLR support is ready and stable
it should changed to opt-out and be enabled by default along with ASLR.
Each application Makefile uses opt-out to ensure that ASLR will be enabled by
default in new directories when the system is compiled with PIE/ASLR. [2]
Mark known build failures as NO_PIE for now.
The only known runtime failure was rtld.
[1] http://www.bsdcan.org/2014/schedule/events/452.en.html
Submitted by: Shawn Webb <lattera@gmail.com>
Discussed between: des@ and Shawn Webb [2]
and MK_LLDB=no, so set those explicitly (now that we can do
that). Simplify tests for these variables as well, since we know they
will always be defined regardless of the phase of the build.
