Commit Graph

496 Commits

Author SHA1 Message Date
Bryan Drewery
497e80911e Remove the old depend (mkdep) code and make FAST_DEPEND the one true way.
Reviewed by:	emaste, hselasky (partial), brooks (brief)
Discussed on:	arch@
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D5742
2016-03-30 23:50:23 +00:00
Warner Losh
bb52d711d0 Use the newly minted Makefile.libcompat to implement libsoft libraries
for the armv6 ABI switch. This also make WITH_LIBSOFT functional on
the arm platform. As a transition thing, this seems to work even
without switching the ABI (we basically build the same libraries
twice when MK_LIBSOFT=yes until the ABI cut over next
month). MK_LIBSOFT remains default no.
2016-03-12 23:25:05 +00:00
Bryan Drewery
dfb898bee2 Avoid MK_TESTS error on stable/10 by just preventing SUBDIR recursion. 2016-03-11 19:28:43 +00:00
Bryan Drewery
abaf3e4ee2 Avoid bmake upgrade NO_MAN warning by just setting MAN to empty.
Suggested by:	imp
2016-03-11 19:24:15 +00:00
Bryan Drewery
7b526e77f2 Revert r296645 as it breaks stable/10->head builds. 2016-03-11 17:21:27 +00:00
Bryan Drewery
e32a2cbe39 FAST_DEPEND: Use .dinclude to enable full .depend logic in bmake.
The inclusion of .MAKE.DEPENDFILE (.depend) has special logic in make
to ignore stale/missing dependencies.  bmake 20160220 added a '.dinclude'
directive that uses the special logic for .depend when including the file.

This fixes a build error when a file is moved or deleted that exists in a
.depend.OBJ file.  This happened in r292782 when sha512c.c "moved" and an
incremental build of lib/libmd would fail with:
  make: don't know how to make /usr/src/lib/libcrypt/../libmd/sha512c.c. Stop

Now this will just be seen as a stale dependency and cause a rebuild:
  make: /usr/obj/usr/src/lib/libmd/.depend.sha512c.o, 13: ignoring stale .depend for /usr/src/lib/libcrypt/../libmd/sha512c.c
  --- sha512c.o ---
  ...
This rebuild will only be done once since the .depend.sha512c.o will
be updated on the build with the -MF flags.

This also removes -MP being passed for the .depend.OBJ generation (which
would create fake targets for system headers) since the logic is no
longer needed to protect from missing files.

Sponsored by:	EMC / Isilon Storage Division
2016-03-11 04:09:56 +00:00
Bryan Drewery
b88b232738 Fix bmake upgrade NO_MAN warnings.
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-03-11 04:09:53 +00:00
Bryan Drewery
125297980a Fix make -n upgrade_checks.
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-03-11 04:09:47 +00:00
Bryan Drewery
0481f7852d Don't ever create object directories here with MK_AUTO_OBJ.
Sponsored by:	EMC / Isilon Storage Division
2016-03-08 21:26:25 +00:00
Bryan Drewery
89660f21db Add order for installworld/installkernel.
MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2016-02-24 22:27:25 +00:00
Bryan Drewery
22944b4134 Support a WANT_MAKE_VERSION.
This will be used soon for .dinclude support in FAST_DEPEND.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:18:49 +00:00
Baptiste Daroussin
d29df4d060 Fix make universe when running with non POSIX/C locales using a locale sensitive
pattern

Reported by:	many
Submitted by:	jilles
2016-02-10 00:26:01 +00:00
Enji Cooper
71b7fa1252 Simplify running the FreeBSD test suite
Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.

`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.

Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.

Document `make check` and `make checkworld` in build(7).

Other minor changes:

- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
  `make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
  not found

The MFC is [partly] contingent on other build related changes being MFCed.

Differential Revision: https://reviews.freebsd.org/D4406
MFC after: 2 months
X-MFC to: stable/10
Relnotes: yes
Reviewed by: bdrewery, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division
2016-02-07 18:40:04 +00:00
Bryan Drewery
83b79794a0 Add order for installworld/distribution.
Sponsored by:	EMC / Isilon Storage Division
2016-02-02 20:50:06 +00:00
Ian Lepore
94d8b152b9 Add make universe targets "kernels" and "worlds".
"make kernels" is now shorthand for "make universe -DMAKE_JUST_KERNELS"
"make worlds"  is now shorthand for "make universe -DMAKE_JUST_WORLDS"

The kernels target includes modules (unless you add -DNO_MODULES).

And of course you can still add all the other universe options, such as
"make kernels TARGETS=arm" to build kernels for all arm arches, or
TARGET_ARCH=armv6 to build all armv6 kernels, etc.

Reviewed by:	imp
2016-01-31 17:32:58 +00:00
Bryan Drewery
63c612de72 Always try to upgrade to bmake if not already using it.
This is mostly targetting stable/10 which requires bmake to build and
has issues upgrading from <10.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
PR:		198062
2016-01-07 19:37:11 +00:00
Bryan Drewery
0e87e3cb6f Revert r293286. It was not intended to come in yet. 2016-01-07 00:20:47 +00:00
Bryan Drewery
3ba0785a74 Move the MAKEOBJDIRPREFIX value guard to sys.mk and expand to MAKEOBJDIR.
This will ensure that the variable was not set as a make override, in
make.conf, src.conf or src-env.conf.  It allows setting the value in
src-env.conf when using WITH_AUTO_OBJ since that case properly handles
changing .OBJDIR (except if MAKEOBJDIRPREFIX does not yet exist which is
being discussed to be changed).

This change allows setting a default MAKEOBJDIRPREFIX via local.sys.env.mk.

Sponsored by:	EMC / Isilon Storage Division
2016-01-07 00:19:03 +00:00
Ed Maste
43aa90e584 Update advice on obj disk space requirements
The Makefile is not the best place for this information, but at least it
is now less out of date.
2015-12-14 19:20:18 +00:00
Ruslan Bukin
1fdcc5e5c0 Start support for the RISC-V 64-bit architecture developed by UC Berkeley.
RISC-V is a new ISA designed to support computer research and education, and
is now become a standard open architecture for industry implementations.

This is a minimal set of changes required to run 'make kernel-toolchain'
using external (GNU) toolchain.

The FreeBSD/RISC-V project home: https://wiki.freebsd.org/riscv.

Reviewed by:	andrew, bdrewery, emaste, imp
Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
Differential Revision:	https://reviews.freebsd.org/D4445
2015-12-11 22:55:23 +00:00
Bryan Drewery
b67490a3b0 Fix errors being ignored in many phases of the build since the bmake integration.
Say it with me, "I will not chain commands with && in Makefiles"

This was originally fixed and explained quite well by bde@ in r36074.  The
initial bmake integration caused 'set -e' to stop being used which lead to
r252419.  Later 'set -e' expectations were fixed with bmake in r254980.

Because of the && here, errors would be ignored when building in parallel and
a dependency failed.  Such as bootstrap-tools since it builds everything in
parallel.  If any tool failed in obj/depend/all, it would just ignore the error
and continue to build.  This later would result in cascaded errors that only
confused the real issue.  This could also cause commands after the failed
command to still execute, leading to more confusion.

This should be fine if the command is in a sub-shell such as: (cmd1 && cmd2)

This reverts r252419.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-12-01 19:00:43 +00:00
Simon J. Gerraty
948f327ee4 Rename META_MODE option to DIRDEPS_BUILD
This allows META_FILES option to be renamed META_MODE.
Also add META_COOKIE_TOUCH for use in targets that can benefit
from a cookie when in meta mode.

Differential Revision:	https://reviews.freebsd.org/D4153
Reviewed by:	bdrewery
2015-11-14 03:24:48 +00:00
Bryan Drewery
2753ba1ab6 Move 'make cleandir' from Makefile to Makefile.inc1.
This leads the way for fixing cross-build cleanup, and eventually replacing
'cleandir' with it during the build.

Sponsored by:	EMC / Isilon Storage Division
MFC after:	2 weeks
2015-11-09 23:36:46 +00:00
Bryan Drewery
9160419c7a Add built-in ccache build support via WITH_CCACHE_BUILD option.
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build.  This is explained in
more detail upstream [1] [2].  It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times.  When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'.  Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports.  This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead.  ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss.  For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct.  Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it.  Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings.  GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual.  Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization.  There is also a memcached backend available [5].  Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout.  A
possible working layout is as follows:
  Source: /some/prefix/src1
  Source: /some/prefix/src2
  Source: /some/prefix/src3
  Objdir: /some/prefix/obj
  Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative.  Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files and ccache objects.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
  x buildworld-before
  + buildworld-ccache-empty
  * buildworld-ccache-full
  % buildworld-ccache-full-fastdep
  # buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |%            *                               #                                +|
  |%            *                               #                                +|
  |%            *                               #             xxx                +|
  |                                                           |A                  |
  |                                                                              A|
  |             A                                                                 |
  |A                                                                              |
  |                                             A                                 |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3          4519       4525.04       4520.73       4521.59     3.1104823
  Difference at 95.0% confidence
          758.027 +/- 43.4565
          20.1412% +/- 1.15466%
          (Student's t, pooled s = 19.1726)
  *   3       1823.08        1827.2       1825.62        1825.3     2.0785572
  Difference at 95.0% confidence
          -1938.26 +/- 43.298
          -51.5007% +/- 1.15045%
          (Student's t, pooled s = 19.1026)
  %   3       1266.96       1279.37       1270.47     1272.2667     6.3971113
  Difference at 95.0% confidence
          -2491.3 +/- 44.3704
          -66.1952% +/- 1.17895%
          (Student's t, pooled s = 19.5758)
  #   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-ccache-empty
  * buildkernel-ccache-empty-fastdep
  % buildkernel-ccache-full
  # buildkernel-ccache-full-fastdep
  @ buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |#                        @   %                  *                              |
  |#                        @   %                  *     x                      + |
  |#                        @   %                  *     xx                     ++|
  |                                                      MA                       |
  |                                                                             MA|
  |                                                A                              |
  |                             A                                                 |
  |A                                                                              |
  |                         A                                                     |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        727.97        731.91        728.06     729.31333     2.2492295
  Difference at 95.0% confidence
          156.88 +/- 4.17129
          27.4058% +/- 0.728695%
          (Student's t, pooled s = 1.84034)
  *   3         527.1        528.29        528.08     527.82333    0.63516402
  Difference at 95.0% confidence
          -44.61 +/- 2.33254
          -7.79305% +/- 0.407478%
          (Student's t, pooled s = 1.02909)
  %   3         400.4        401.05        400.62        400.69     0.3306055
  Difference at 95.0% confidence
          -171.743 +/- 2.16453
          -30.0023% +/- 0.378128%
          (Student's t, pooled s = 0.954969)
  #   3        201.94        203.34        202.28        202.52    0.73020545
  Difference at 95.0% confidence
          -369.913 +/- 2.40293
          -64.6212% +/- 0.419774%
          (Student's t, pooled s = 1.06015)
  @   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR:		182944 [4]
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
Relnotes:	yes
2015-11-08 00:50:18 +00:00
Bryan Drewery
56585ab576 Rework the 'make -n -n' feature such that '-n' recurses and '-N' does not.
Bmake has a documented feature of '-N' to skip executing commands which is
specifically intended for debugging top-level builds and not recursing into
sub-directories.  This matches the older 'make -n' behavior we added which made
'-n -n' the recursing target and '-n' a non-recursing target.

Removing the '-n -n' feature allows the build to work as documented in
the bmake manpage with '-n' and '-N'.  The older '-n -n' feature was also
not documented anywhere that I could see.

Note that the ${_+_} var is still needed as currently bmake incorrectly
executes '+' commands when '-N' is specified.

The '-n' and '-n -n' features were broken for several reasons prior to this.
r251748 made '_+_' never expand with '-n -n' which resulted in many
sub-directories not being visited until fixed 2 years later in r288391, and
many targets were given .MAKE over the past few years which resulted in
non-sub-make commands, such as rm and ln and mtree, to be executed.

This should also allow removing some indirection hacks in bsd.subdir.mk and
other cases of .USE that have a .MAKE by using '+'.

Sponsored by:	EMC / Isilon Storage Division
Discussed on:	arch@ (mostly silence)
2015-10-17 16:42:54 +00:00
Bryan Drewery
e07b2be5e9 Tweak the guard more to suggest 'all' if SUBDIR_OVERRIDE is specified. In that
case 'all' does make sense.

MFC after:	2 weeks
X-MFC-With:	r289411
Sponsored by:	EMC / Isilon Storage Division
2015-10-16 20:59:03 +00:00
Bryan Drewery
824f98d3fb Tweak the default target to not suggest 'all' since it really doesn't do
anything useful for most users.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-10-16 05:59:59 +00:00
Bryan Drewery
ba831a2cb7 Consider top-level targets to be .PHONY as bmake won't build them
otherwise if a file with the same name is found in the directory.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-10-15 05:32:56 +00:00
Baptiste Daroussin
2bcf1d422b Add installconfig as a top level target 2015-10-09 22:14:07 +00:00
Bryan Drewery
fa5e9cbbd7 Fix the .MAKE added in r251750 to properly support the historical -n -n.
The condition used matches the condition in sys.mk for setting _+_ to blank
or +.

With this -n will continue to not descend into Makefile.inc1, while -n -n will
and cause Makefile.inc1's target to run with -n.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-09-29 18:57:30 +00:00
Bryan Drewery
a87f765b19 Rework r287900 to keep arm64/aarch64 stable in the TARGETS list.
This is relevant for makeman using the 'make targets' output in src.conf(5).

This makes a _UNIVERSE_TARGETS that removes arm64 if the build
requirements are not met.
2015-09-17 04:54:49 +00:00
Bryan Drewery
f80c5b811b Get arm64/aarch64 into 'make targets' output to fix makeman. 2015-09-17 04:43:07 +00:00
Simon J. Gerraty
1c9b3241f2 Fix generation of src.conf.5
Since makeman turns all options on, we need to guard somethings from
make(showconfig)
2015-07-03 06:10:09 +00:00
Simon J. Gerraty
65faab7f4a targets now needs .PHONY 2015-06-22 16:13:17 +00:00
Andrew Turner
d667f11204 Move the universe done dependency on the worlds completing out of the
loop, we only need to add it once.

Obtained from:	ABT Systems Ltd
2015-06-16 21:29:46 +00:00
Andrew Turner
0b5f5ef2cd Remove unneeded dependencies, these are now handled through the _done
target.

Obtained from:	ABT Systems Ltd
2015-06-16 21:11:31 +00:00
Andrew Turner
1272c4aad2 Add a new target universe_${target}_done to print the completion message.
Without this we could print this message in the wrong place when building
with MAKE_JUST_WORLDS is set.

Obtained from:	ABT Systems Ltd
2015-06-16 19:23:34 +00:00
Andrew Turner
149da95c87 Add a universe_${target}_worlds target to simplify the logic to find when
to start building kernels.

Obtained from:	ABT Systems Ltd
2015-06-16 18:43:08 +00:00
Warner Losh
35c9a2a9a1 A more compatible fix to MK_META_MODE not being defined. Also, encase
bmake specific constructs not needed for make bootstrap so fmake
doesn't see them. This works with fmake just well enough for us to
build bmake to build the rest of the tree without fatal errors. Tested
only with fmake package.
2015-06-16 17:55:20 +00:00
Andrew Turner
6cdaf31e2a Add a comment to the end of the world and kernel cases in the universe
target to help follow the make magic.

Obtained from:	ABT Systems Ltd
2015-06-16 17:27:53 +00:00
Craig Rodrigues
054fbb2f15 If MK_META_MODE is unset, assume a value of no.
This is needed to build HEAD on FreeBSD 10.1,
which has bmake, but does not have
/usr/share/mk/src.opts.mk

Reviewed by: sjg
Reported by: jenkins
2015-06-14 22:36:27 +00:00
Simon J. Gerraty
1fc3d968ac Since sys.mk now handles META_MODE and META_FILES options
we need not worry about them here.
Checking for .PARSEDIR no longer needed.
2015-06-10 13:08:45 +00:00
Simon J. Gerraty
98e0ffaefb Merge sync of head 2015-05-27 01:19:58 +00:00
Ed Maste
504f34b362 Don't add arm64 to universe builds if the user provided a TARGETS list
Differential Revision:	https://reviews.freebsd.org/D2375
Reported by:	andrew
Reviewed by:	andrew, imp
Sponsored by:	The FreeBSD Foundation
2015-04-28 17:13:05 +00:00
Enji Cooper
3f29603d1b Fix a typo (linnker -> linker) 2015-04-27 18:21:42 +00:00
Ed Maste
f384739d96 Add arm64 to universe if binutils is available.
arm64 relies on an external binutils port or package right now, because
the in-tree linker from binutils 2.17.50 does not support arm64. Add
arm64 to universe if the linker is available. If not output a message
that arm64 is skipped.

buildworld and buildkernel use the external binutils automatically, so
it's sufficient to run 'pkg install aarch64-binutils' to build
FreeBSD/arm64.

Differential Revision:	https://reviews.freebsd.org/D2302
Reviewed by:	andrew, imp
Sponsored by:	The FreeBSD Foundation
2015-04-16 22:35:19 +00:00
Warner Losh
28e1e3e250 Make TARGET_ARCH=powerpc64 work without TARGET=powerpc. 2015-03-21 16:54:01 +00:00
Andrew Turner
8daa81674e Start to import support for the AArch64 architecture from ARM. This change
only adds support for kernel-toolchain, however it is expected further
changes to add kernel and userland support will be committed as they are
reviewed.

As our copy of binutils is too old the devel/aarch64-binutils port needs
to be installed to pull in a linker.

To build either TARGET needs to be set to arm64, or TARGET_ARCH set to
aarch64. The latter is set so uname -p will return aarch64 as existing
third party software expects this.

Differential Revision:	https://reviews.freebsd.org/D2005
Relnotes:	Yes
Sponsored by:	The FreeBSD Foundation
2015-03-19 13:53:47 +00:00
Simon J. Gerraty
aa9bd11d5d If we are doing META_MODE use targets/Makefile as top-level
since this one isn't suitable.
2014-11-30 18:50:15 +00:00
Simon J. Gerraty
9268022b74 Merge from head@274682 2014-11-19 01:07:58 +00:00