This is not properly respecting WITHOUT or ARCH dependencies in target/.
Doing so requires a massive effort to rework targets/ to do so. A
better approach will be to either include the SUBDIR Makefiles directly
and map to DIRDEPS or just dynamically lookup the SUBDIR. These lose
the benefit of having a userland/lib, userland/libexec, etc, though and
results in a massive package. The current implementation of targets/ is
very unmaintainable.
Currently rescue/rescue and sys/modules are still not connected.
Sponsored by: EMC / Isilon Storage Division
- Support more of the toolchain from TOOLSDIR.
- This also improves 'make bootstrap-tools' to pass, for example,
AS=/usr/bin/as to Makefile.inc1, which will tell cross-tools to use
external toolchain support and avoid building things we won't be using
in the build.
- Always set the PATH to contain the staged TOOLSDIR directories when
not building the bootstrap targets.
The previous version was only setting this at MAKE.LEVEL==0 and if the
TOOLSDIR existed. Both of these prevented using staged tools that were
built during the build though as DIRDEPS with .host dependencies, such
as the fix for needing usr.bin/localedef.host in r291311.
This is not a common tool so we must build and use it during the build,
and need to be prepared to change PATH as soon as it appears.
This should also fix the issue of host dependencies disappearing from
Makefile.depend and then reappearing due to the start of the fresh build not
having the directory yet, resulting in the tools that were built not actually
being used.
- Only use LEGACY_TOOLS while building in Makefile.inc1. After r291317
and r291546 there is no need to add LEGACY_TOOLS into the PATH for
the pseudo/targets/toolchain build.
- Because the pseudo/targets/toolchain will now build its own
[clang-]tblgen, the special logic in clang.build.mk is no longer needed.
- LEGACY_TOOLS is no longer used outside of targets/pseudo/bootstrap-tools
so is no longer passed into the environment in its build.
Sponsored by: EMC / Isilon Storage Division
Doing this causes more trouble than it is worth regarding cyclic
dependencies. It should not be needed after cleaning up MACHINE=host
builds in r291324.
Sponsored by: EMC / Isilon Storage Division
This is mostly working around the converts/iconv port having '../ces/file.o'
in its OBJS list which resulted in '.depend../ces/file.o'. Now it will have
'.depend.._ces_file.o'.
Other implementations have :T which would result in '.depend.file.o' here, but
that could lead to collisions.
X-MFC-With: r291554
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
-MP creates empty targets for all dependency files, which can be useful when a
dependency is deleted from the file system. This would otherwise cause an
error for "don't know how to build FOO" since the .depend file is included
with the dependency registered.
This is mostly a workaround for the misc/dahdi-kmod port using '::' for one of
its dependencies, while -MP uses just ':'. This results in an 'Inconsistent
operator for' error.
X-MFC-With: r290433
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
- Add a kvaddr_type to represent kernel virtual addresses instead of
unsigned long.
- Add a struct kvm_nlist which is a stripped down version of struct nlist
that uses kvaddr_t for n_value.
- Add a kvm_native() routine that returns true if an open kvm descriptor
is for a native kernel and memory image.
- Add a kvm_open2() function similar to kvm_openfiles(). It drops the
unused 'swapfile' argument and adds a new function pointer argument for
a symbol resolving function. Native kernels still use _fdnlist() from
libc to resolve symbols if a resolver function is not supplied, but cross
kernels require a resolver.
- Add a kvm_nlist2() function similar to kvm_nlist() except that it uses
struct kvm_nlist instead of struct nlist.
- Add a kvm_read2() function similar to kvm_read() except that it uses
kvaddr_t instead of unsigned long for the kernel virtual address.
- Add a new kvm_arch switch of routines needed by a vmcore backend.
Each backend is responsible for implementing kvm_read2() for a given
vmcore format.
- Use libelf to read headers from ELF kernels and cores (except for
powerpc cores).
- Add internal helper routines for the common page offset hash table used
by the minidump backends.
- Port all of the existing kvm backends to implement a kvm_arch switch and
to be cross-friendly by using private constants instead of ones that
vary by platform (e.g. PAGE_SIZE). Static assertions are present when
a given backend is compiled natively to ensure the private constants
match the real ones.
- Enable all of the existing vmcore backends on all platforms. This means
that libkvm on any platform should be able to perform KVA translation
and read data from a vmcore of any platform.
Tested on: amd64, i386, sparc64 (marius)
Differential Revision: https://reviews.freebsd.org/D3341
headers.
This resulted in 'don't know how to make .o.' errors after the changes in
r289286.
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
This is going to be used to allow DIRDEPS to be bootstrapped off of
LIBADD/DPADD. It currently works for internal libraries which have a
DIR defined for them but also use the .a library from a src-mapped obj
directory. It can also be useful for using -L without a --sysroot per
LIBADD to use the OBJDIR version of the libraries.
I didn't review every LIBADD, so it is possible this is missing some.
Sponsored by: EMC / Isilon Storage Division
We need to not use -nostdinc since it breaks building of clang itself. Use
-isystem rather than -I/usr/include and -nostdinc which gets us using
the stage include directory before searching the real host headers.
This allows removing more of the -I hacks to get host headers since the
headers are no longer excluded. The -B seemed unneeded.
This fixes building of secure/lib/libcrypto which was looking at the
/usr/include/openssl/asn1.h header rather than the staged one.
This fixes building of clang which wants to find its own internal
headers in the STAGEDIR/usr/lib/clang/* path.
Sponsored by: EMC / Isilon Storage Division
The main problem was bitrot after elftoolchain being swapped in for the
GNU toolchain.
This also reworks how the list of 'host allowed' libraries is determined
to only allow INTERNALLIBs, which is needed for libelftc to come in.
For usr.bin/readelf use the same hack, as libelf and libdward, to bring in
the needed sys/ headers for host builds. This has not yet been a problem due
to readelf not being built as a host tool in buildworld. This is possible
in the meta build though when building the toolchain.
Sponsored by: EMC / Isilon Storage Division
Tracking these leads to situations where meta mode will consider the
file to be out of date if /bin/sh or /bin/ln are newer than the source
file. There's no reason for meta mode to do this as make is already
handling the rebuild dependency fine.
Sponsored by: EMC / Isilon Storage Division
This was finding libraries that were installed into DESTDIR/usr/lib,
where DESTDIR is the stage directory, and then adding in usr/lib to
DIRDEPS. Just exclude the STAGE_ROOT if defined.
Discussed with: sjg
Sponsored by: EMC / Isilon Storage Division
This has caused much confusion for myself as there are quite a lot of
variables that depend on having a proper ${.OBJDIR}.
Sponsored by: EMC / Isilon Storage Division
This both avoids some dependencies on xinstall.host and allows
bootstrapping on older releases to work due to lack of at least 'install -l'
support.
Sponsored by: EMC / Isilon Storage Division
This is only for sys/boot. INTERNALPROG is not a build tool, it is just a way
to generate OBJS from a list of SRCS and use those objects elsewhere.
Sponsored by: EMC / Isilon Storage Division
Keep old CPUTYPEs around for compatibility. Also include
a list of typical values for FreeBSD.
# Split out from other changes in D4155
Differential Revision: https://reviews.freebsd.org/D4155
has not ran yet.
This fixes building objects directly, or skipping 'make depend', not generating
required headers first. This case did work without FAST_DEPEND so there's no
reason it should not work here as well.
An example of this can be seen building in gnu/usr.bin/binutils/libbfd
without running 'make depend' first to generate config.h.
Sponsored by: EMC / Isilon Storage Division
MFC after: 3 weeks
X-MFC-With: r290433
This allows most of the build to simply consider MK_META_MODE
Update to latest dirdeps.mk so we can do:
make -f dirdeps.mk bin/cat.i386
Reviewed by: bdrewery
This allows META_FILES option to be renamed META_MODE.
Also add META_COOKIE_TOUCH for use in targets that can benefit
from a cookie when in meta mode.
Differential Revision: https://reviews.freebsd.org/D4153
Reviewed by: bdrewery
This also fixes truncating the path that the links were installed to, which
was most likely going to be the same directory the library was in anyhow.
Let bsd.sys.mk handle SYMLINKS via stage_symlinks. stage_libs continues to
handle the SHLIB_LINK though since it is not a SYMLINKS.
This fixes a race, seen easily in lib/libthr, where libpthread_p.a is created
by both stage_libs and stage_symlinks resulting in 'ln: File exists'.
Sponsored by: EMC / Isilon Storage Division
Discussed with: sjg
Discussing with sjg@, we agree the better fix is to be done in meta.stage.mk.
This also broke staging of SYMLINKS for non-shared libraries, such as for
lib/libcompiler_rt, which results in all Makefile.depend removing it.
Sponsored by: EMC / Isilon Storage Division
meta.stage.mk is handling ${SYMLINKS:T} for stage_libs already. The logic in
bsd.sys.mk to handle ${SYMLINKS} was brought in r247817 when it was moved out
of bsd.prog.mk and bsd.lib.mk into bsd.sys.mk. The logic previously was
limited to bsd.prog.mk.
This fixes a race, seen easily in lib/libthr, where libpthread_p.a is created
by both stage_libs and stage_symlinks resulting in 'ln: File exists'.
Sponsored by: EMC / Isilon Storage Division
centralizes the handling of CC and HOST_CC.
This fixes a bug with WITH_CCACHE_BUILD when using MACHINE=host since
CC is overridden in local.init.mk via src.opts.mk long before bsd.compiler.mk
is included.
Originally the ccache implementation was placed in local.init.mk but moved
to bsd.compiler.mk as it seemed more proper and avoided other ordering
issues.
Sponsored by: EMC / Isilon Storage Division
Similar to the original reason for these dependency hints to be added,
in r124637, the missing-dependency file case can lead to building of the
wrong source.
A clear example of this is in gnu/lib/libstdc++ where the .PATH contains
both contrib/gcc and contrib/libstdc++/src.
contrib/gcc has a debug.c.
contrib/libstdc++/src has a debug.cc.
When building for the objects of debug.o, debug.So, and debug.po, it is
ambiguous for which src file to use due to the suffix transformation
rules, even though the proper one is listed first in .PATH.
This was normally avoided due to these dependency hints for the initial
build, and then mkdep would add an explicit 'debug.o: debug.cc'
dependency into the .depend file. WITH_FAST_DEPEND does not generate
the .depend file with these, but puts them into .depend.debug.o instead.
Rather than extending the exists() check to each object's .depend.*
file, just enable the hint when when using WITH_FAST_DEPEND. It fixes
the problem and seems to be safe enough to use since it is mapping SRCS
back to OBJS, rather than letting make make assumptions from OBJS to
SRCS.
A similar check mapping objects to headers is present in some mk files
but was not extended here for FAST_DEPEND since it has not yet been
found to be a problem.
X-MFC-With: r290433
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build. This is explained in
more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times. When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'. Stats are provided at the end of this message.
This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.
The option name was picked to match the existing option in ports. This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.
Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead. ccache does nothing special in these modes, although there is
no harm in calling it for them.
CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss. For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct. Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.
The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it. Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings. GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.
Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual. Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization. There is also a memcached backend available [5]. Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout. A
possible working layout is as follows:
Source: /some/prefix/src1
Source: /some/prefix/src2
Source: /some/prefix/src3
Objdir: /some/prefix/obj
Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative. Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.
distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.
The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).
Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.
The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files and ccache objects.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16
The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.
buildworld:
x buildworld-before
+ buildworld-ccache-empty
* buildworld-ccache-full
% buildworld-ccache-full-fastdep
# buildworld-fastdep
+-------------------------------------------------------------------------------+
|% * # +|
|% * # +|
|% * # xxx +|
| |A |
| A|
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 4519 4525.04 4520.73 4521.59 3.1104823
Difference at 95.0% confidence
758.027 +/- 43.4565
20.1412% +/- 1.15466%
(Student's t, pooled s = 19.1726)
* 3 1823.08 1827.2 1825.62 1825.3 2.0785572
Difference at 95.0% confidence
-1938.26 +/- 43.298
-51.5007% +/- 1.15045%
(Student's t, pooled s = 19.1026)
% 3 1266.96 1279.37 1270.47 1272.2667 6.3971113
Difference at 95.0% confidence
-2491.3 +/- 44.3704
-66.1952% +/- 1.17895%
(Student's t, pooled s = 19.5758)
# 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)
buildkernel:
x buildkernel-before
+ buildkernel-ccache-empty
* buildkernel-ccache-empty-fastdep
% buildkernel-ccache-full
# buildkernel-ccache-full-fastdep
@ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|# @ % * |
|# @ % * x + |
|# @ % * xx ++|
| MA |
| MA|
| A |
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 727.97 731.91 728.06 729.31333 2.2492295
Difference at 95.0% confidence
156.88 +/- 4.17129
27.4058% +/- 0.728695%
(Student's t, pooled s = 1.84034)
* 3 527.1 528.29 528.08 527.82333 0.63516402
Difference at 95.0% confidence
-44.61 +/- 2.33254
-7.79305% +/- 0.407478%
(Student's t, pooled s = 1.02909)
% 3 400.4 401.05 400.62 400.69 0.3306055
Difference at 95.0% confidence
-171.743 +/- 2.16453
-30.0023% +/- 0.378128%
(Student's t, pooled s = 0.954969)
# 3 201.94 203.34 202.28 202.52 0.73020545
Difference at 95.0% confidence
-369.913 +/- 2.40293
-64.6212% +/- 0.419774%
(Student's t, pooled s = 1.06015)
@ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)
[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30
PR: 182944 [4]
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
Relnotes: yes
This is because the .meta files generated from filemon already contain a
list of all files read to generate the object.
X-MFC-With: r290433
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
This is especially noticeable in the kernel obj directory since it
includes so many files.
X-MFC-With: r290433
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
This speeds up buildworld by 16% on my system and buildkernel by 35%.
Rather than calling mkdep(1), which is just a wrapper around 'cc -E',
use the modern -MD -MT -MF flags to gather and generate dependencies during
compilation. This flag was introduced in GCC "a long time ago", in GCC 3.0,
and is also supported by Clang. (It appears that ICC also supports this but I
do not have access to test it). This avoids running the preprocessor *twice*
for every build, in both 'make depend' and 'make all'. This is especially
noticeable when using ccache since it does not cache preprocessor results from
mkdep(1) / 'cc -E', but still speeds up compilation with the -MD flags.
For 'make depend' a tree-walk is still done to ensure that all DPSRCS
are generated when expected, and that beforedepend/afterdepend and
_EXTRADEPEND are all still respected. In time this may change but for now
I've been conservative. The time for a tree-walk with -j combined with
SUBDIR_PARALLEL is not significant. For example, it takes about 9 seconds
with -j15 to walk all of src/ for 'make depend' now on my system.
A .depend file is still generated with the various rules that apply to
the final target, or custom rules. Otherwise there are now
per-built-object-file .depend files, such as .depend.filename.o. These
are included directly by make rather than populating .depend with a loop
and .depend lines, which only added overhead to the now almost-NOP 'make
depend' phase.
Before this I experimented with having mkdep(1) called in parallel per-file.
While this improved the kernel and lib/libc 'make depend' phase, it resulted
in slower build times overall.
The -M flags are removed from CFLAGS when linking since they have no effect.
Enabling this by default, for src or out-of-src, can be done once more testing
has been done, such as a ports exp-run, and with more compilers.
The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_FAST_DEPEND=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16
buildworld:
x buildworld-before
+ buildworld-fastdep
+-------------------------------------------------------------------------------+
|+ |
|+ |
|+ xx x|
| |_MA___||
|A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)
buildkernel:
x buildkernel-before
+ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|+ x |
|++ xx|
| A||
|A| |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)
Sponsored by: EMC / Isilon Storage Division
MFC after: 3 weeks
Relnotes: yes
