Afterbuild target allows to perform operations on fully built binary.
This is needed to allow for ELF feature flags modification during
world build.
Submitted by: Dawid Gorecki <dgr@semihalf.com>
Reviewed by: imp
Obtained from: Semihalf
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D29551
This option has been equivalent to any form of C++ support since libstdc++
was removed. Therefore, replace all MK_LIBCPLUSPLUS uses with MK_CXX.
Reviewed By: emaste
Differential Revision: https://reviews.freebsd.org/D27974
As suggested in D27598. This also supports MK_WERROR.clang=no and
MK_WERROR.gcc=no to support the existing NO_WERROR.<compiler> uses.
Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D27601
There are two options:
- WITH_INIT_ALL_ZERO: Zero all variables on the stack.
- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns.
The exact pattern are a compiler implementation detail and vary by type.
They are somewhat documented in the LLVM commit message:
https://reviews.llvm.org/rL349442
I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather
than naming them after the LLVM specific compiler flags.
In a range of consumer products, options like these are used in
both debug and production builds with debugs builds using patterns
(intended to provoke crashes on use of uninitialized values) and
production using zeros (deemed more likely to lead to harmless
misbehavior or NULL-pointer dereferences).
Reviewed by: emaste
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D27131
-development is long and awkward, and is also inconsistent with prior art
from the Linux world, which uses -dev (Debian) or -devel (Red Hat). Follow
the Debian convention, and similarly for debug info packages.
Also remove redundant pkgbase development tag from includes. We already tag
include files with package=runtime,dev; there is no need to separately tag
them as dev.
Discussed with: bapt
Reviewed by: manu
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D24139
bsd.cpu.mk is included by bsd.init.mk before bsd.linker.mk, so it
was always setting the flag since LINKER_FEATURES wasn't defined.
Reported by: mhorne
Reviewed by: imp, mhorne
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D23076
Replace explicit TARGET_* variables with COMPAT_* versions defined based
on where the file is being included.
Also, require that bsd.compat.mk be included directly. It's not going to
be widely used so always loading it in bsd.prog.mk doesn't make sense.
Instead users can include it directly.
Reviewed by: imp, bdrewery (prior revision)
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22059
Linkage is controlled by two make knobs:
WANT_COMPAT - Prefer to link against the compat ABI.
NEED_COMPAT - Link against the compat ABI or fail to build.
Supported values are "32", "soft", and "any". The latter meaning pick
the first[0] supported compat ABI.
This can be used to provide test binaries for compat ABIs or to link
ABI-specific programs.
[0] We currently support only one compat ABI at a time, but this may
change in the future and some code in this commit is structured to ease
that change.
Reviewed by: bdrewery, jhb
Obtained from: CheriBSD (in concept)
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22023
GCC uses "dynamic" TLS models when -fpic or -fPIC is explicitly
specified on the command line (which is only true for shared libraries).
It uses "static" (or "exec") TLS models otherwise. In particular, GCC
does _not_ use dynamic TLS models when PIC is implicitly enabled (which
it is on MIPS), only if a PIC flag is explicitly provided.
llvm uses "dynamic" TLS models if PIC is enabled either via a PIC flag
or if it is implicily enabled (as on MIPS64). This means that llvm on
MIPS64 always uses "dynamic" TLS models. However, dynamic TLS models
do not work for static binaries and libraries as the __tls_get_addr
function they invoke is only defined in rtld.
Written by: jhb
Reviewed by: arichardson
Differential Revision: https://reviews.freebsd.org/D21699
The default package use to be FreeBSD-runtime but it should only contain
binaries and libs enough to boot to single user and repair the system, it
is also very handy to have a package that can be tranform to a small mfsroot.
So create a new package named FreeBSD-utilities and make it the default one.
Also move a few binaries and lib into this package when it make sense.
Reviewed by: bapt, gjb
Differential Revision: https://reviews.freebsd.org/D21506
Building binaries as PIE allows the executable itself to be loaded at a
random address when ASLR is enabled (not just its shared libraries).
With this change PIE objects have a .pieo extension and INTERNALLIB
libraries libXXX_pie.a.
MK_PIE is disabled for some kerberos5 tools, Clang, and Subversion, as
they explicitly reference .a libraries in their Makefiles. These can
be addressed on an individual basis later. MK_PIE is also disabled for
rtld-elf because it is already position-independent using bespoke
Makefile rules.
Currently only dynamically linked binaries will be built as PIE.
Discussed with: dim
Reviewed by: kib
MFC after: 1 month
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18423
The linker's -z now flag sets the DF_BIND_NOW flag, which signals to the
runtime loader that all relocation processing should be performed at
process startup rather than on demand. In combination with lld's
default of enabling relro this causes the GOT to be made read-only when
the process starts, preventing straightforward GOT overwrite attacks.
Shawn Webb discovered a failure on HardenedBSD with BIND_NOW and ifunc
use, which resulted in my rtld fix in r340137. Add a BIND_NOW knob as
it is trivial to do so and is a useful ELF hardening feature. This
change is equivalent to HardenedBSD's but not identical as there are
other diffs/conflicts nearby.
Note that our ELF Tool Chain readelf does not currently decode the
DF_BIND_NOW flag - see PR232983.
Reviewed by: brooks
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17846
warning:
make[3]: "/usr/src/share/mk/bsd.prog.mk" line 274: warning: duplicate
script for target "_scriptsinstall" ignored
make[3]: "/usr/src/share/mk/bsd.prog.mk" line 274: warning: using
previous script for "_scriptsinstall" defined here
Reviewed by: kevans
It was an old TRE that had plenty of bugs and no performance gain over
regex(3). I disabled it by default in r323615, and there was some confusion
about what the knob does- likely due to poor naming on my part- to the tune
of "well, it sounds like it should speed things up" (mentioned by multiple
people).
To compound this, I have no intention of maintaining a second regex
implementation. If someone would like to step up and volunteer to maintain a
lean-and-mean implementation for grep, this is OK, but we have very few
volunteers to maintain even our primary regex implementation.
Now that OBJS has grown an OBJS_SRCS_FILTER variable, use this variable
in the computation of BCOBJS and LLOBJS too. Also move BCOBJS and LLOBJS
computation to be next to the OBJS computation: this should both make
the parallel structure clearer and serve to remind people changing OBJS
that parallel changes are required in BCOBJS and LLOBJS.
A side effect of this change is that BCOBJS and LLOBJS will be available
even when LLVM_LINK has not been defined, but that seems like a positive
change: there's no reason we can't ask "what bitcode files would you
generate" just because we can't link those files together into a
complete bitcode representation of a binary or library.
Reviewed by: sjg
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D12701
The build rule describing how to create ${PROG_FULL}.{bc,ll} is only
dependent on LLVM_LINK being defined, not on MK_DEBUG_FILES being "yes".
Move the addition of ${PROG_FULL}.{bc,ll} out of the conditional block
under `.if ${MK_DEBUG_FILES} != "no"` and up next to where the build
rules for ${PROG_FULL}.{bc,ll} are defined.
Reviewed by: emaste
MFC after: 1 week
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D12703
Some makefiles do reachover builds.
In some cases it is convenient to list subdirs of the distribution
in SRCS.
It is not very convenient, or always even desirable to have corresponding
subdirs in .OBJDIR, so OBJS_SRCS_FILTER allows the makefile to choose.
The default value 'R' matches existing practice.
But a makefile can set OBJS_SRCS_FILTER= T (the R gets added by
bsd.init.mk) to avoid the need for subdirs in .OBJDIR
Differential Revision: https://reviews.freebsd.org/D12218
Reviewed by: bdrewery
- Include debug symbols in static libraries. This permits binaries
to include debug symbols for functions obtained from static libraries.
- Permit the C/C++ compiler flags added for MK_DEBUG_FILES to be
overridden by setting DEBUG_FILES_CFLAGS. Use this to limit the debug
information for llvm libraries and binaries.
Reviewed by: emaste
Sponsored by: DARPA / AFRL
Differential Revision: https://reviews.freebsd.org/D12025
== Rationale ==
r295380 introduced "make check" and consolidated means for running
test code in an attempt to simplify running tests. One could either
install files/libraries/programs and run "make check", or run "make check"
with an explicit CHECKDIR, e.g., `make check CHECKDIR=$(make -V.OBJDIR)``.
One criticism that was received is that "make check" should be run with
the intent of making dev->test->commit easier, which means that the target
audience's workflow should be developers. One developer pattern available
in other opensource projects is to run test code from a developer sandbox,
instead of installing to a system.
== Method ==
This approach is slightly different from the standard approach, in the sense
that it builds and installs into a deterministic directory under .OBJDIR (as I call it,
the "sandbox"), then runs "make check" against that. In the event the test
run is successful, the deterministic directory is removed to save space.
== Approach ==
bsd.lib.mk, bsd.prog.mk:
To support this functionality, a new variable `HAS_TESTS` is being added.
HAS_TESTS enables appropriate behavior with bsd.lib.mk and bsd.prog.mk, as
follows:
- Add "make check" as an available target from the directory.
- Pass down appropriate variables via ${TESTS_ENV}, i.e.,
${TESTS_LD_LIBRARY_PATH} and ${TESTS_PATH}.
One should add "HAS_TESTS" to directories containing tests in them, e.g. from
bin/sh/Makefile,
HAS_TESTS=
SUBDIR.${MK_TESTS}+= tests
HAS_TESTS doesn't automatically add the tests subdirectory for flexibility
reasons.
bsd.opts.mk, src.opts.mk:
- The knob ${MK_MAKE_CHECK_USE_SANDBOX} has been added, both to explicitly
direct (internally) when to set a deterministic ${DESTDIR} and to also allow
users to disable this behavior globally, i.e., via src.conf.
- MK_TESTS has been promoted from src.opts.mk to bsd.opts.mk to leverage
syntactic sugar for having MK_TESTS be a dependency for
MK_MAKE_CHECK_USE_SANDBOX, but to also ensure that src.opts.mk isn't required
to use suite.test.mk (which is a dependency of bsd.test.mk).
suite.test.mk:
- beforecheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified
from a no-op to:
-- Build.
-- Run "make hierarchy" on the sandbox dir.
-- Install the tests/files to the sandbox dir.
- aftercheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified
from a no-op to:
-- Remove the sandbox dir.
Again, because the dependency order set in bsd.test.mk is
beforecheck -> check -> aftercheck, "make check" will not be run unless
"beforecheck" completes successfully, and "aftercheck" will not be run unless
"beforecheck" and "check" complete successfully.
== Caveats ==
- This target must either be run with MK_INSTALL_AS_USER or as root. Otherwise
it will fail when running "make install" as the default user/group for many
makefiles when calling INSTALL is root/wheel.
- This target must be run from a suitable top-level directory. For example,
running tests from `tests/sys/fs/tmpfs` won't work, but `tests/sys/fs` will,
because `tests/sys/fs/tmpfs` relies on files installed by `tests/sys/fs`.
- Running MK_INSTALL_AS_USER may introduce determinism issues. However, using
it could identify deficiences in tests in terms of needing to be run as
root, which are not properly articulated in the test requirements.
- The doesn't negate the need for running "make installworld" and
"make checkworld", etc. Again, this just is intended to simplify the
dev->test->commit workflow.
== Cleanup done ==
- CHECKDIR is removed; one can use "MK_MAKE_CHECK_USE_SANDBOX=no" to enable
"legacy" (r295380) behavior.
MFC after: 2 months
Relnotes: yes (CHECKDIR removed; "make check" behavior changed)
Requested by: jhb
Reviewed by: arch (silence), testing (silence)
Differential Revision: D11905
Running `make libfoo.ll` or `make libfoo.bc` within a library directory
will now give us an LLVM IR version of the library, and `make foo.full.ll`
or `make foo.full.bc` will give us an IR version of a binary.
As part of this change, we add an LLVM_LINK variable to sys.mk that can be
specified/overridden using an external toolchain.
Reviewed by: bdrewery, brooks
Approved by: rwatson (mentor)
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D8388
kgzldr.o is installed into /usr/lib but using bsd.prog.mk. Add
/usr/lib to the base system directory list so that debug files are
installed into /usr/lib/debug/usr/lib, not /usr/lib/.debug .
Approved by: re (gjb)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
The DPADD data in .depend will be redundant with what is in the .meta file.
Also extend NO_EXTRADEPEND support to bsd.prog.mk.
Approved by: re (blanket, META_MODE)
Sponsored by: EMC / Isilon Storage Division
This breaks cross-building with WITH_META_MODE since it will rebuild
'build-tools' during the 'everything' phase.
A more proper fix is coming to bmake to implicitly require .META unless
.NOMETA (and other restrictions) are in place.
Adding .META to targets-to-build will ensure that they will rebuild if there
is no .meta file.
Adding it to all SUFFIXES and objects ensures that at least objects will
rebuild if there is no .meta file.
This will be reverted if bmake's behavior changes to rebuild on missing .meta
files.
Sponsored by: EMC / Isilon Storage Division
Currently the base.txz distribution does not get the BSD.debug.dist mtree
extracted into it. So if you start from that and then try to build a 3rd-party
application outside of buildworld it will by-default try installing the
debug files into a missing directory if they are being installed into /usr/lib.
Check for the existence before forcing the directory to be created rather than
the older way of running a shell command with test -d || mkdir -p always.
Reported by: HardenedBSD (https://github.com/HardenedBSD/secadm/issues/23)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5411
This was a regression in r295985.
bsd.dep.mk adds to SRCS for dtrace probes, yacc grammars and some
others.
The code that is moving is planned to be removed once FAST_DEPEND is
default (and the only option) though since FAST_DEPEND doesn't use this.
Pointyhat to: bdrewery
Sponsored by: EMC / Isilon Storage Division
This allows 'make analyze' or 'make OBJ.clang-analyzer' to run the
Clang static analyzer and present results on stdout.
Obtained from: NetBSD (CVS Rev. 1.3)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5449
r96164 added them to avoid recursing twice with _SUBDIR. That issue was
fixed in bsd.subdir.mk in r291635 for all targets.
Sponsored by: EMC / Isilon Storage Division
Rather than depend on .depend not existing, check the actual
.depend.OBJ file that will be used for that object. If it doesn't
exist then use the guessed dependencies.
FAST_DEPEND may never have a .depend file. Not having one means all of the
previous logic would over-depend all object files on all headers which is not
what we wanted. It also means that if a .depend is generated before a build
is done for _EXTRADEPEND (such as for PROG or LIB) then all of these
dependencies would not be used since the .depend wasn't generated from mkdep
and the real .depend.* files are not generated until the build.
Sponsored by: EMC / Isilon Storage Division
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build. This is explained in
more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times. When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'. Stats are provided at the end of this message.
This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.
The option name was picked to match the existing option in ports. This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.
Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead. ccache does nothing special in these modes, although there is
no harm in calling it for them.
CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss. For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct. Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.
The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it. Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings. GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.
Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual. Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization. There is also a memcached backend available [5]. Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout. A
possible working layout is as follows:
Source: /some/prefix/src1
Source: /some/prefix/src2
Source: /some/prefix/src3
Objdir: /some/prefix/obj
Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative. Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.
distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.
The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).
Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.
The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files and ccache objects.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16
The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.
buildworld:
x buildworld-before
+ buildworld-ccache-empty
* buildworld-ccache-full
% buildworld-ccache-full-fastdep
# buildworld-fastdep
+-------------------------------------------------------------------------------+
|% * # +|
|% * # +|
|% * # xxx +|
| |A |
| A|
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 4519 4525.04 4520.73 4521.59 3.1104823
Difference at 95.0% confidence
758.027 +/- 43.4565
20.1412% +/- 1.15466%
(Student's t, pooled s = 19.1726)
* 3 1823.08 1827.2 1825.62 1825.3 2.0785572
Difference at 95.0% confidence
-1938.26 +/- 43.298
-51.5007% +/- 1.15045%
(Student's t, pooled s = 19.1026)
% 3 1266.96 1279.37 1270.47 1272.2667 6.3971113
Difference at 95.0% confidence
-2491.3 +/- 44.3704
-66.1952% +/- 1.17895%
(Student's t, pooled s = 19.5758)
# 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)
buildkernel:
x buildkernel-before
+ buildkernel-ccache-empty
* buildkernel-ccache-empty-fastdep
% buildkernel-ccache-full
# buildkernel-ccache-full-fastdep
@ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|# @ % * |
|# @ % * x + |
|# @ % * xx ++|
| MA |
| MA|
| A |
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 727.97 731.91 728.06 729.31333 2.2492295
Difference at 95.0% confidence
156.88 +/- 4.17129
27.4058% +/- 0.728695%
(Student's t, pooled s = 1.84034)
* 3 527.1 528.29 528.08 527.82333 0.63516402
Difference at 95.0% confidence
-44.61 +/- 2.33254
-7.79305% +/- 0.407478%
(Student's t, pooled s = 1.02909)
% 3 400.4 401.05 400.62 400.69 0.3306055
Difference at 95.0% confidence
-171.743 +/- 2.16453
-30.0023% +/- 0.378128%
(Student's t, pooled s = 0.954969)
# 3 201.94 203.34 202.28 202.52 0.73020545
Difference at 95.0% confidence
-369.913 +/- 2.40293
-64.6212% +/- 0.419774%
(Student's t, pooled s = 1.06015)
@ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)
[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30
PR: 182944 [4]
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
Relnotes: yes
