Commit Graph

608 Commits

Author SHA1 Message Date
Ian Lepore
1f14b5e4c0 Add support for "make universe_kernels -DMAKE_GENERIC_KERNELS" to build
just the GENERIC kernels for each arch (including variations such as
GENERIC-NODEBUG, GENERIC64, etc).

This helps with quickly doing a test build for all[*] arches without
building dozens of variant kernels for the arches that have lots of
hardware/board/system variations.

[*] Not all arches have a generic kernel (but they probably should for
test-building purposes, even if it can't boot on any real hardware).
2017-06-12 17:22:23 +00:00
Bryan Drewery
1a9cc831dd META_MODE: Add framework to force rebuilding for major ABI changes.
Normally META_MODE ignores host files for "meta mode" decisions on whether a
file should be rebuilt or not.  This is because a simple installworld can
update timestamps and cause the next build to rebuild all host tools, when the
previous ones may not have any changes in the source tree.  These tools are
normally still ABI compatible.  They are only rebuilt if NO_META_IGNORE_HOST is
set from the workaround/hack in r301467.

One of the major problems with this is when a host tool has objects spread
across many revisions that have mixed-ABI.  For example, if struct stat were to
change on the host, some objects for a tool may have different ideas of that
struct's definition.  If just 1 source file were modified and rebuilt and
linked into the tool, then that toll will have mixed-ABI objects and crash.
This exact thing happened with the ino64 commit in r301467 followed by a
trivial update to libbfd in r318750.  The resulting binary would crash in
buildworld.

Sponsored by:	Dell EMC Isilon
2017-06-05 05:17:39 +00:00
Bryan Drewery
848d5e929b Cache compiler metadata and reuse it at installworld time.
Right after cross-tools, a compiler-metadata.mk file is created that
stores all of the bsd.compiler.mk metadata.  It is then read in
with a fail-safe during installworld time.

The file is explicitly removed when invoking cross-tools to ensure that
a stale file is not left around from odd manual 'make _cross-tools' ->
'make installworld' invocations.

This fixes several issues:
    - With WITH_SYSTEM_COMPILER (default yes on head and no on releng/11.0):
      If you build on a system where the bootstrap compiler does not
      build due to the host compiler matching the in-tree one, but then
      installworld on another system where that logic fails (a
      bootstrap compiler is needed), the installworld immediately fails
      with:
           sh: cc: not found
      Note that fixing this logic may then hit a case where a rebuild is
      attempted in installworld.  Normally cc would be ran with
      'CFLAGS+=ERROR-tried-to-rebuild-during-make-install' to cause an
      error such as:
          cc: error: no such file or directory: 'ERROR-tried-to-rebuild-during-make-install'
      However, now it will just fail with the 'cc: not found' error.
      Inspection of the compile line will show
      'ERROR-tried-to-rebuild-during-make-install';  It's not useful to
      set CC to anything other than 'cc' during install as it is more
      helpful to see the attempted compile rather than some other bogus
      error.
    - This now avoids running bsd.compiler.mk (cc executions) even more
      during installworld.  There are compiler-dependent SUBDIR in the
      tree which required having a compiler during install.

There is at least 1 case where CC is still executed in the install,
such as from a LOOKUP!= in secure/lib/libcrypto/Makefile.inc checking
for 'vzeroall' support.  This is not significant for installworld
as the lookup has a fallback (and hides its error) and only modifies CFLAGS,
thus it's not worth fixing.

PR:		212877
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-04-13 22:07:34 +00:00
Ruslan Bukin
6d7d1e1ed8 Include RISC-V target to universe build.
Check if RISC-V external toolchain package is installed,
otherwise skip build.

Reviewed by:	emaste
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D10344
2017-04-12 10:45:19 +00:00
Ed Maste
1b766820b0 Unconditionally include arm64 in make universe / tinderbox
As of r316629 FreeBSD/arm64 uses the in-tree LLD linker by default, and
does not require an external an aarch64-binutils port or package.

Sponsored by:	The FreeBSD Foundation
2017-04-08 16:14:30 +00:00
Yoshihiro Takahashi
2b375b4edd Remove pc98 support completely.
I thank all developers and contributors for pc98.

Relnotes:	yes
2017-01-28 02:22:15 +00:00
Ruslan Bukin
7804dd5212 Add full softfloat and hardfloat support for RISC-V.
Hardfloat is now default (use riscv64sf as TARGET_ARCH
for softfloat).

Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D8529
2016-11-16 15:21:32 +00:00
Ruslan Bukin
5bca221511 Add full softfloat and hardfloat support for MIPS.
This adds new target architectures for hardfloat:
mipselhf mipshf mips64elhf mips64hf.

Tested in QEMU only.

Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
Differential Revision:	https://reviews.freebsd.org/D8376
2016-10-31 15:33:58 +00:00
Justin Hibbits
dc9b124d66 Create a new MACHINE_ARCH for Freescale PowerPC e500v2
Summary:
The Freescale e500v2 PowerPC core does not use a standard FPU.
Instead, it uses a Signal Processing Engine (SPE)--a DSP-style vector processor
unit, which doubles as a FPU.  The PowerPC SPE ABI is incompatible with the
stock powerpc ABI, so a new MACHINE_ARCH was created to deal with this.
Additionaly, the SPE opcodes overlap with Altivec, so these are mutually
exclusive.  Taking advantage of this fact, a new file, powerpc/booke/spe.c, was
created with the same function set as in powerpc/powerpc/altivec.c, so it
becomes effectively a drop-in replacement.  setjmp/longjmp were modified to save
the upper 32-bits of the now-64-bit GPRs (upper 32-bits are only accessible by
the SPE).

Note: This does _not_ support the SPE in the e500v1, as the e500v1 SPE does not
support double-precision floating point.

Also, without a new MACHINE_ARCH it would be impossible to provide binary
packages which utilize the SPE.

Additionally, no work has been done to support ports, work is needed for this.
This also means no newer gcc can yet be used.  However, gcc's powerpc support
has been refactored which would make adding a powerpcspe-freebsd target very
easy.

Test Plan:
This was lightly tested on a RouterBoard RB800 and an AmigaOne A1222
(P1022-based) board, compiled against the new ABI.  Base system utilities
(/bin/sh, /bin/ls, etc) still function appropriately, the system is able to boot
multiuser.

Reviewed By:	bdrewery, imp
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D5683
2016-10-22 01:57:15 +00:00
Bryan Drewery
9e2f435fc1 Add a 'make print-dir' that simply traverses all directories and prints them.
This is useful for finding connected directories.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-08-31 19:30:52 +00:00
Bryan Drewery
c8de0c57d7 Avoid showing the bootstrap make command for check-old, etc.
Reported by:	koobs
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-08-12 13:52:51 +00:00
Bryan Drewery
6597f1f81f make world: Allow installworld to be ran in parallel.
This has been safe for a while.

Sponsored by:	EMC / Isilon Storage Division
2016-08-08 18:13:03 +00:00
Bryan Drewery
b17f5b2261 SYSTEM_COMPILER: Rework the logic to allow a 'make test-system-compiler'.
1. Always calculate what the expected values are.
2. Add 'make test-system-compiler' to show all of the computed values
   vs the wanted values.
3. Extend the .info line to buildkernel/kernel-toolchain/toolchain/_cross-tools.
4. Consolidate all of the logic to one condition.

MFC after:	3 days
Sponsored by:	EMC / Isilon Storage Division
2016-07-24 18:05:05 +00:00
Bryan Drewery
b987685fe0 Revert r302670 and r302671 for now.
MACHINE_CPUARCH smells like MACHINE except for arm64/aarch64 which
has it backwards.
2016-07-13 01:35:53 +00:00
Bryan Drewery
8a6bdc134d Create one list of replacements for MACHINE_CPUARCH as MACHINE_CPUARCH_SUB.
This also adds missing s/aarch64/arm64 to the sys.mk version and also
adds back armv6hf for universe since it was added to the sys.mk version
in r300438.

MFC after:	3 days
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D7159
2016-07-12 18:57:25 +00:00
Bryan Drewery
e5bf62aac7 WITH_META_MODE: Whitelist 'make kernel-toolchain'
Approved by:	re (blanket, META_MODE)
Sponsored by:	EMC / Isilon Storage Division
2016-06-22 19:07:41 +00:00
Bryan Drewery
9c8809d77f Fix top-level builds with fmake / 9.x.
Reported by:	allanjude
Approved by:	re (marius)
Sponsored by:	EMC / Isilon Storage Division
2016-06-21 20:14:32 +00:00
Bryan Drewery
8f8c247740 WITH_META_MODE: Whitelist 'make kernel' and 'make world'.
installkernel is technically META_MODE safe but doesn't need an explicit
approval to use it since it's all disabled via .PHONY.

world uses 'make -B installworld' which already will disable META_MODE
via the -B (.MAKE.MODE=compat) usage.

Approved by:	re (implicit)
Sponsored by:	EMC / Isilon Storage Division
2016-06-15 23:58:06 +00:00
Bryan Drewery
d97607185a tinderbox/universe: Only show arm64 warning if it was in the TARGETS list.
This was a flaw in my change in r287903 but also in the original change
in r282156 since it used empty(${TARGETS}) rather than empty(TARGETS).

Reported by:	lidl
Approved by:	re (gjb)
Sponsored by:	EMC / Isilon Storage Division
2016-06-15 23:58:00 +00:00
Bryan Drewery
3a3594e584 Fix makeman showing dependency of DIRDEPS_BUILD->META_MODE.
This broke in r301887 with the meta mode whitelist.  'make showconfig'
still needs WITH_META_MODE support.

Approved by:	re (implicit)
Sponsored by:	EMC / Isilon Storage Division
2016-06-14 18:37:33 +00:00
Bryan Drewery
b6d5c4f594 Fix build from stable/10 with fmake.
This was broken in r301888.

fmake does not look in share/mk by default and thus does not yet
have MK_META_MODE set with default.

Pointyhat to:	bdrewery
Approved by:	re (implicit)
Sponsored by:	EMC / Isilon Storage Division
2016-06-14 17:23:22 +00:00
Bryan Drewery
b8ef21b88a WITH_META_MODE: Lessen the filemon(4) requirement scope.
- Move the sys.mk filemon requirement to bsd.init.mk as a warning.
  This is intended only to show when building directly in a subdirectory
  without filemon loaded.
- Move the error into Makefile and only apply it when building
  from the META_TGT_WHITELIST target list.

-DNO_FILEMON can be used to suppress both the warning and the error but
makes WITH_META_MODE less useful.  It will only compare build commands
in this mode rather than track all dependencies.

This fixes installing from a jail which doesn't need filemon in this
phase [1].

Reported by:	Nikolai Lifanov <lifanov@mail.lifanov.com> [1]
Approved by:	re (implicit)
Sponsored by:	EMC / Isilon Storage Division
2016-06-14 16:20:19 +00:00
Bryan Drewery
2c6fc926b3 WITH_META_MODE: Whitelist targets that are meta-mode-safe.
META_TGT_WHITELIST is added to define which build targets are safe for
meta mode.  See comments for more details.

This fixes 'make delete-old-libs' to properly show the interactive
prompt.

Approved by:	re (implicit)
Sponsored by:	EMC / Isilon Storage Division
2016-06-14 16:20:14 +00:00
Bryan Drewery
341d14b404 Add more missing .PHONY
Approved by:	re (implicit)
Sponsored by:	EMC / Isilon Storage Division
2016-06-14 16:20:08 +00:00
Bryan Drewery
6f4b2e7f5c Only include bsd.compiler.mk for bmakes that use -m .../share/mk by default.
Otherwise they error when trying to include the file, which also has
its own dependencies.

Sponsored by:	EMC / Isilon Storage Division
2016-06-05 23:05:17 +00:00
Bryan Drewery
3681768c3f WITH_META_MODE: Enable bmake's missing meta rebuild feature 2016-06-05 23:04:42 +00:00
Bryan Drewery
bd2464674d Fix bmake version upgrade logic to use the new bmake.
Otherwise it was only used on the next build.

This was a flaw in r295980.

MFC after:	3 days
Sponsored by:	EMC / Isilon Storage Division
2016-06-05 23:04:15 +00:00
Warner Losh
2c0e9e2a09 Make armv6 hard float abi by default. Kill armv6hf.
Allow CPUTYPE=soft to build the current soft-float abi libraries.
Add UPDATING entry to announce this.

Approved by: re@ (gjb)
2016-05-18 06:01:18 +00:00
Bryan Drewery
f62610d7bb Follow-up r298220: Don't pass down META_MODE which will still enable it.
Reported by:	Nikolai Lifanov <lifanov@mail.lifanov.com>
Sponsored by:	EMC / Isilon Storage Division
2016-04-18 18:39:43 +00:00
Bryan Drewery
d7698c0722 META_MODE: Disable during installworld and similar.
META_MODE may create cookies during staging of files to WORLDTMP that would
also prevent installation of the files to the final DESTDIR, since the cookie
already exists.  This is not yet the case but will be soon.  Prevent other
similar issues by disabling META_MODE for any top-level install targets.

Sponsored by:	EMC / Isilon Storage Division
2016-04-18 18:14:05 +00:00
Glen Barber
0edd2576c0 MFH
Sponsored by:	The FreeBSD Foundation
2016-04-16 02:32:12 +00:00
Bryan Drewery
a561b1060c Mark some more .PHONY targets.
Sponsored by:	EMC / Isilon Storage Division
2016-04-14 21:04:37 +00:00
Bryan Drewery
9b8f7eaf6a Define the *soft targets properly.
Sponsored by:	EMC / Isilon Storage Division
2016-04-14 19:29:35 +00:00
Glen Barber
d60840138f MFH
Sponsored by:	The FreeBSD Foundation
2016-04-04 23:55:32 +00:00
Bryan Drewery
497e80911e Remove the old depend (mkdep) code and make FAST_DEPEND the one true way.
Reviewed by:	emaste, hselasky (partial), brooks (brief)
Discussed on:	arch@
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D5742
2016-03-30 23:50:23 +00:00
Glen Barber
538354481e MFH
Sponsored by:	The FreeBSD Foundation
2016-03-14 18:54:29 +00:00
Warner Losh
bb52d711d0 Use the newly minted Makefile.libcompat to implement libsoft libraries
for the armv6 ABI switch. This also make WITH_LIBSOFT functional on
the arm platform. As a transition thing, this seems to work even
without switching the ABI (we basically build the same libraries
twice when MK_LIBSOFT=yes until the ABI cut over next
month). MK_LIBSOFT remains default no.
2016-03-12 23:25:05 +00:00
Bryan Drewery
dfb898bee2 Avoid MK_TESTS error on stable/10 by just preventing SUBDIR recursion. 2016-03-11 19:28:43 +00:00
Bryan Drewery
abaf3e4ee2 Avoid bmake upgrade NO_MAN warning by just setting MAN to empty.
Suggested by:	imp
2016-03-11 19:24:15 +00:00
Bryan Drewery
7b526e77f2 Revert r296645 as it breaks stable/10->head builds. 2016-03-11 17:21:27 +00:00
Bryan Drewery
e32a2cbe39 FAST_DEPEND: Use .dinclude to enable full .depend logic in bmake.
The inclusion of .MAKE.DEPENDFILE (.depend) has special logic in make
to ignore stale/missing dependencies.  bmake 20160220 added a '.dinclude'
directive that uses the special logic for .depend when including the file.

This fixes a build error when a file is moved or deleted that exists in a
.depend.OBJ file.  This happened in r292782 when sha512c.c "moved" and an
incremental build of lib/libmd would fail with:
  make: don't know how to make /usr/src/lib/libcrypt/../libmd/sha512c.c. Stop

Now this will just be seen as a stale dependency and cause a rebuild:
  make: /usr/obj/usr/src/lib/libmd/.depend.sha512c.o, 13: ignoring stale .depend for /usr/src/lib/libcrypt/../libmd/sha512c.c
  --- sha512c.o ---
  ...
This rebuild will only be done once since the .depend.sha512c.o will
be updated on the build with the -MF flags.

This also removes -MP being passed for the .depend.OBJ generation (which
would create fake targets for system headers) since the logic is no
longer needed to protect from missing files.

Sponsored by:	EMC / Isilon Storage Division
2016-03-11 04:09:56 +00:00
Bryan Drewery
b88b232738 Fix bmake upgrade NO_MAN warnings.
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-03-11 04:09:53 +00:00
Bryan Drewery
125297980a Fix make -n upgrade_checks.
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-03-11 04:09:47 +00:00
Glen Barber
7d536dc855 MFH
Sponsored by:	The FreeBSD Foundation
2016-03-10 21:16:01 +00:00
Bryan Drewery
0481f7852d Don't ever create object directories here with MK_AUTO_OBJ.
Sponsored by:	EMC / Isilon Storage Division
2016-03-08 21:26:25 +00:00
Glen Barber
52259a98ad MFH
Sponsored by:	The FreeBSD Foundation
2016-03-02 16:14:46 +00:00
Bryan Drewery
89660f21db Add order for installworld/installkernel.
MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2016-02-24 22:27:25 +00:00
Bryan Drewery
22944b4134 Support a WANT_MAKE_VERSION.
This will be used soon for .dinclude support in FAST_DEPEND.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:18:49 +00:00
Glen Barber
3a3f435f98 Include a 'package-pkg' target, intended for use for
architectures we do not provide upstream pkg(8) packages.

This is not tied to anything as-is, and likely will break
your system if used (based on experience with testing with
powerpc).

There is an overwhelming amount of evil happening here,
so until the issues are fixed, it will not be tied into the
'packages' target.

Sponsored by:	The FreeBSD Foundation
2016-02-24 03:05:05 +00:00
Glen Barber
30922917c8 MFH
Sponsored by:	The FreeBSD Foundation
2016-02-10 04:20:39 +00:00
Baptiste Daroussin
d29df4d060 Fix make universe when running with non POSIX/C locales using a locale sensitive
pattern

Reported by:	many
Submitted by:	jilles
2016-02-10 00:26:01 +00:00
Glen Barber
a5395435ff Add logic to rotate the package repository, keyed on PKG_VERSION,
and create a 'latest' symlink to the PKG_VERSION repository path.

Suggested by:	des
Sponsored by:	The FreeBSD Foundation
2016-02-09 19:27:38 +00:00
Dag-Erling Smørgrav
ea85204f94 Rename the signpackages target to sign-packages, and include it in the
packages meta-target so 'make packages' now does everything.
2016-02-08 16:34:13 +00:00
Dag-Erling Smørgrav
ee1627c4db Split the packages target into stage-packages and create-packages to make
it possible to roll new packages from an existing build without having to
restage them.
2016-02-08 16:08:13 +00:00
Glen Barber
bbb51924bb MFH
Sponsored by:	The FreeBSD Foundation
2016-02-08 12:16:01 +00:00
Enji Cooper
71b7fa1252 Simplify running the FreeBSD test suite
Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.

`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.

Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.

Document `make check` and `make checkworld` in build(7).

Other minor changes:

- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
  `make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
  not found

The MFC is [partly] contingent on other build related changes being MFCed.

Differential Revision: https://reviews.freebsd.org/D4406
MFC after: 2 months
X-MFC to: stable/10
Relnotes: yes
Reviewed by: bdrewery, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division
2016-02-07 18:40:04 +00:00
Glen Barber
221b349912 MFH
Sponsored by:	The FreeBSD Foundation
2016-02-02 22:27:48 +00:00
Bryan Drewery
83b79794a0 Add order for installworld/distribution.
Sponsored by:	EMC / Isilon Storage Division
2016-02-02 20:50:06 +00:00
Ian Lepore
94d8b152b9 Add make universe targets "kernels" and "worlds".
"make kernels" is now shorthand for "make universe -DMAKE_JUST_KERNELS"
"make worlds"  is now shorthand for "make universe -DMAKE_JUST_WORLDS"

The kernels target includes modules (unless you add -DNO_MODULES).

And of course you can still add all the other universe options, such as
"make kernels TARGETS=arm" to build kernels for all arm arches, or
TARGET_ARCH=armv6 to build all armv6 kernels, etc.

Reviewed by:	imp
2016-01-31 17:32:58 +00:00
Glen Barber
3d98f473dc Add 'stagekernel' target, which invokes 'distributekernel'
with -DNO_ROOT to create the METALOG mtree(8) file.

Separate the default STAGEDIR for world (WSTAGEDIR) and kernel
(KSTAGEDIR).

Fix the 'create-kernel-packages' target to work properly.

Evaluate if 'kernel' is set when invoking mtree-to-plist.awk,
which splits the kernel and kernel.debug into separate plist
files.

Fix METALOG creation when building/packaging multiple kernels.

Sponsored by:	The FreeBSD Foundation
2016-01-27 21:10:03 +00:00
Glen Barber
d46f04328b Add a 'signpackages' target, which creates the pkg repository
metadata files and uses the path of PKGSIGNKEY for signing,
if set.

Sponsored by:	The FreeBSD Foundation
2016-01-25 16:18:08 +00:00
Glen Barber
e60680024a MFH
Sponsored by:	The FreeBSD Foundation
2016-01-12 01:23:45 +00:00
Bryan Drewery
63c612de72 Always try to upgrade to bmake if not already using it.
This is mostly targetting stable/10 which requires bmake to build and
has issues upgrading from <10.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
PR:		198062
2016-01-07 19:37:11 +00:00
Bryan Drewery
0e87e3cb6f Revert r293286. It was not intended to come in yet. 2016-01-07 00:20:47 +00:00
Bryan Drewery
3ba0785a74 Move the MAKEOBJDIRPREFIX value guard to sys.mk and expand to MAKEOBJDIR.
This will ensure that the variable was not set as a make override, in
make.conf, src.conf or src-env.conf.  It allows setting the value in
src-env.conf when using WITH_AUTO_OBJ since that case properly handles
changing .OBJDIR (except if MAKEOBJDIRPREFIX does not yet exist which is
being discussed to be changed).

This change allows setting a default MAKEOBJDIRPREFIX via local.sys.env.mk.

Sponsored by:	EMC / Isilon Storage Division
2016-01-07 00:19:03 +00:00
Glen Barber
b626f5a73a MFH r289384-r293170
Sponsored by:	The FreeBSD Foundation
2016-01-04 19:19:48 +00:00
Ed Maste
43aa90e584 Update advice on obj disk space requirements
The Makefile is not the best place for this information, but at least it
is now less out of date.
2015-12-14 19:20:18 +00:00
Ruslan Bukin
1fdcc5e5c0 Start support for the RISC-V 64-bit architecture developed by UC Berkeley.
RISC-V is a new ISA designed to support computer research and education, and
is now become a standard open architecture for industry implementations.

This is a minimal set of changes required to run 'make kernel-toolchain'
using external (GNU) toolchain.

The FreeBSD/RISC-V project home: https://wiki.freebsd.org/riscv.

Reviewed by:	andrew, bdrewery, emaste, imp
Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
Differential Revision:	https://reviews.freebsd.org/D4445
2015-12-11 22:55:23 +00:00
Bryan Drewery
b67490a3b0 Fix errors being ignored in many phases of the build since the bmake integration.
Say it with me, "I will not chain commands with && in Makefiles"

This was originally fixed and explained quite well by bde@ in r36074.  The
initial bmake integration caused 'set -e' to stop being used which lead to
r252419.  Later 'set -e' expectations were fixed with bmake in r254980.

Because of the && here, errors would be ignored when building in parallel and
a dependency failed.  Such as bootstrap-tools since it builds everything in
parallel.  If any tool failed in obj/depend/all, it would just ignore the error
and continue to build.  This later would result in cascaded errors that only
confused the real issue.  This could also cause commands after the failed
command to still execute, leading to more confusion.

This should be fine if the command is in a sub-shell such as: (cmd1 && cmd2)

This reverts r252419.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-12-01 19:00:43 +00:00
Simon J. Gerraty
948f327ee4 Rename META_MODE option to DIRDEPS_BUILD
This allows META_FILES option to be renamed META_MODE.
Also add META_COOKIE_TOUCH for use in targets that can benefit
from a cookie when in meta mode.

Differential Revision:	https://reviews.freebsd.org/D4153
Reviewed by:	bdrewery
2015-11-14 03:24:48 +00:00
Bryan Drewery
2753ba1ab6 Move 'make cleandir' from Makefile to Makefile.inc1.
This leads the way for fixing cross-build cleanup, and eventually replacing
'cleandir' with it during the build.

Sponsored by:	EMC / Isilon Storage Division
MFC after:	2 weeks
2015-11-09 23:36:46 +00:00
Bryan Drewery
9160419c7a Add built-in ccache build support via WITH_CCACHE_BUILD option.
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build.  This is explained in
more detail upstream [1] [2].  It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times.  When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'.  Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports.  This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead.  ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss.  For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct.  Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it.  Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings.  GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual.  Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization.  There is also a memcached backend available [5].  Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout.  A
possible working layout is as follows:
  Source: /some/prefix/src1
  Source: /some/prefix/src2
  Source: /some/prefix/src3
  Objdir: /some/prefix/obj
  Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative.  Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files and ccache objects.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
  x buildworld-before
  + buildworld-ccache-empty
  * buildworld-ccache-full
  % buildworld-ccache-full-fastdep
  # buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |%            *                               #                                +|
  |%            *                               #                                +|
  |%            *                               #             xxx                +|
  |                                                           |A                  |
  |                                                                              A|
  |             A                                                                 |
  |A                                                                              |
  |                                             A                                 |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3          4519       4525.04       4520.73       4521.59     3.1104823
  Difference at 95.0% confidence
          758.027 +/- 43.4565
          20.1412% +/- 1.15466%
          (Student's t, pooled s = 19.1726)
  *   3       1823.08        1827.2       1825.62        1825.3     2.0785572
  Difference at 95.0% confidence
          -1938.26 +/- 43.298
          -51.5007% +/- 1.15045%
          (Student's t, pooled s = 19.1026)
  %   3       1266.96       1279.37       1270.47     1272.2667     6.3971113
  Difference at 95.0% confidence
          -2491.3 +/- 44.3704
          -66.1952% +/- 1.17895%
          (Student's t, pooled s = 19.5758)
  #   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-ccache-empty
  * buildkernel-ccache-empty-fastdep
  % buildkernel-ccache-full
  # buildkernel-ccache-full-fastdep
  @ buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |#                        @   %                  *                              |
  |#                        @   %                  *     x                      + |
  |#                        @   %                  *     xx                     ++|
  |                                                      MA                       |
  |                                                                             MA|
  |                                                A                              |
  |                             A                                                 |
  |A                                                                              |
  |                         A                                                     |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        727.97        731.91        728.06     729.31333     2.2492295
  Difference at 95.0% confidence
          156.88 +/- 4.17129
          27.4058% +/- 0.728695%
          (Student's t, pooled s = 1.84034)
  *   3         527.1        528.29        528.08     527.82333    0.63516402
  Difference at 95.0% confidence
          -44.61 +/- 2.33254
          -7.79305% +/- 0.407478%
          (Student's t, pooled s = 1.02909)
  %   3         400.4        401.05        400.62        400.69     0.3306055
  Difference at 95.0% confidence
          -171.743 +/- 2.16453
          -30.0023% +/- 0.378128%
          (Student's t, pooled s = 0.954969)
  #   3        201.94        203.34        202.28        202.52    0.73020545
  Difference at 95.0% confidence
          -369.913 +/- 2.40293
          -64.6212% +/- 0.419774%
          (Student's t, pooled s = 1.06015)
  @   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR:		182944 [4]
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
Relnotes:	yes
2015-11-08 00:50:18 +00:00
Bryan Drewery
56585ab576 Rework the 'make -n -n' feature such that '-n' recurses and '-N' does not.
Bmake has a documented feature of '-N' to skip executing commands which is
specifically intended for debugging top-level builds and not recursing into
sub-directories.  This matches the older 'make -n' behavior we added which made
'-n -n' the recursing target and '-n' a non-recursing target.

Removing the '-n -n' feature allows the build to work as documented in
the bmake manpage with '-n' and '-N'.  The older '-n -n' feature was also
not documented anywhere that I could see.

Note that the ${_+_} var is still needed as currently bmake incorrectly
executes '+' commands when '-N' is specified.

The '-n' and '-n -n' features were broken for several reasons prior to this.
r251748 made '_+_' never expand with '-n -n' which resulted in many
sub-directories not being visited until fixed 2 years later in r288391, and
many targets were given .MAKE over the past few years which resulted in
non-sub-make commands, such as rm and ln and mtree, to be executed.

This should also allow removing some indirection hacks in bsd.subdir.mk and
other cases of .USE that have a .MAKE by using '+'.

Sponsored by:	EMC / Isilon Storage Division
Discussed on:	arch@ (mostly silence)
2015-10-17 16:42:54 +00:00
Bryan Drewery
e07b2be5e9 Tweak the guard more to suggest 'all' if SUBDIR_OVERRIDE is specified. In that
case 'all' does make sense.

MFC after:	2 weeks
X-MFC-With:	r289411
Sponsored by:	EMC / Isilon Storage Division
2015-10-16 20:59:03 +00:00
Bryan Drewery
824f98d3fb Tweak the default target to not suggest 'all' since it really doesn't do
anything useful for most users.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-10-16 05:59:59 +00:00
Glen Barber
324fd1ce05 MFH to r289370
Sponsored by:	The FreeBSD Foundation
2015-10-15 17:36:56 +00:00
Bryan Drewery
ba831a2cb7 Consider top-level targets to be .PHONY as bmake won't build them
otherwise if a file with the same name is found in the directory.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-10-15 05:32:56 +00:00
Baptiste Daroussin
9be27fdc00 Merge from head 2015-10-09 22:45:54 +00:00
Baptiste Daroussin
2bcf1d422b Add installconfig as a top level target 2015-10-09 22:14:07 +00:00
Baptiste Daroussin
5a2b666ce5 Merge from head 2015-10-01 09:36:43 +00:00
Bryan Drewery
fa5e9cbbd7 Fix the .MAKE added in r251750 to properly support the historical -n -n.
The condition used matches the condition in sys.mk for setting _+_ to blank
or +.

With this -n will continue to not descend into Makefile.inc1, while -n -n will
and cause Makefile.inc1's target to run with -n.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-09-29 18:57:30 +00:00
Bryan Drewery
a87f765b19 Rework r287900 to keep arm64/aarch64 stable in the TARGETS list.
This is relevant for makeman using the 'make targets' output in src.conf(5).

This makes a _UNIVERSE_TARGETS that removes arm64 if the build
requirements are not met.
2015-09-17 04:54:49 +00:00
Bryan Drewery
f80c5b811b Get arm64/aarch64 into 'make targets' output to fix makeman. 2015-09-17 04:43:07 +00:00
Baptiste Daroussin
b779ff2b9c Rename kernel-pkgs into create-kernel-packages to be consistent with world 2015-09-12 11:45:10 +00:00
Baptiste Daroussin
b5ff185e19 Merge from head 2015-09-12 11:41:31 +00:00
Simon J. Gerraty
1c9b3241f2 Fix generation of src.conf.5
Since makeman turns all options on, we need to guard somethings from
make(showconfig)
2015-07-03 06:10:09 +00:00
Simon J. Gerraty
65faab7f4a targets now needs .PHONY 2015-06-22 16:13:17 +00:00
Baptiste Daroussin
2fbd60ec47 Merge from head @274131 2015-06-20 00:58:46 +00:00
Andrew Turner
d667f11204 Move the universe done dependency on the worlds completing out of the
loop, we only need to add it once.

Obtained from:	ABT Systems Ltd
2015-06-16 21:29:46 +00:00
Andrew Turner
0b5f5ef2cd Remove unneeded dependencies, these are now handled through the _done
target.

Obtained from:	ABT Systems Ltd
2015-06-16 21:11:31 +00:00
Baptiste Daroussin
baec3daef1 Merge from head@274131 2015-06-16 20:01:01 +00:00
Andrew Turner
1272c4aad2 Add a new target universe_${target}_done to print the completion message.
Without this we could print this message in the wrong place when building
with MAKE_JUST_WORLDS is set.

Obtained from:	ABT Systems Ltd
2015-06-16 19:23:34 +00:00
Andrew Turner
149da95c87 Add a universe_${target}_worlds target to simplify the logic to find when
to start building kernels.

Obtained from:	ABT Systems Ltd
2015-06-16 18:43:08 +00:00
Warner Losh
35c9a2a9a1 A more compatible fix to MK_META_MODE not being defined. Also, encase
bmake specific constructs not needed for make bootstrap so fmake
doesn't see them. This works with fmake just well enough for us to
build bmake to build the rest of the tree without fatal errors. Tested
only with fmake package.
2015-06-16 17:55:20 +00:00
Andrew Turner
6cdaf31e2a Add a comment to the end of the world and kernel cases in the universe
target to help follow the make magic.

Obtained from:	ABT Systems Ltd
2015-06-16 17:27:53 +00:00
Baptiste Daroussin
dad2fb7ece Merge from head 2015-06-15 10:56:01 +00:00
Craig Rodrigues
054fbb2f15 If MK_META_MODE is unset, assume a value of no.
This is needed to build HEAD on FreeBSD 10.1,
which has bmake, but does not have
/usr/share/mk/src.opts.mk

Reviewed by: sjg
Reported by: jenkins
2015-06-14 22:36:27 +00:00
Simon J. Gerraty
1fc3d968ac Since sys.mk now handles META_MODE and META_FILES options
we need not worry about them here.
Checking for .PARSEDIR no longer needed.
2015-06-10 13:08:45 +00:00
Simon J. Gerraty
98e0ffaefb Merge sync of head 2015-05-27 01:19:58 +00:00
Baptiste Daroussin
7757a1b4dc Merge from head 2015-05-03 19:30:11 +00:00