174 Commits

Author SHA1 Message Date
bdrewery
90e73584af META_MODE: Default to the silent build.
Sponsored by:	EMC / Isilon Storage Division
2016-02-26 22:13:56 +00:00
bdrewery
57ff5041e8 Track .meta files for .sh but don't preserve timestamp.
See r291320 for more information.

Sponsored by:	EMC / Isilon Storage Division
2016-02-26 22:13:42 +00:00
bdrewery
c336785ed6 Allow configuring .MAKE.MODE via META_MODE as meta.sys.mk does.
meta.sys.mk is only used for the DIRDEPS_BUILD.

Sponsored by:	EMC / Isilon Storage Division
2016-02-26 22:13:38 +00:00
bdrewery
59cfefb558 META_MODE: Ensure bmake does not use filemon if it is not loaded.
Sponsored by:	EMC / Isilon Storage Division
2016-01-19 22:41:55 +00:00
bdrewery
2beef85d71 Define .MAKE.MODE to normal to avoid the need for :U later.
Sponsored by:	EMC / Isilon Storage Division
2016-01-19 22:41:44 +00:00
bdrewery
14f143093d Revert r293286. It was not intended to come in yet. 2016-01-07 00:20:47 +00:00
bdrewery
469f276a09 Move the MAKEOBJDIRPREFIX value guard to sys.mk and expand to MAKEOBJDIR.
This will ensure that the variable was not set as a make override, in
make.conf, src.conf or src-env.conf.  It allows setting the value in
src-env.conf when using WITH_AUTO_OBJ since that case properly handles
changing .OBJDIR (except if MAKEOBJDIRPREFIX does not yet exist which is
being discussed to be changed).

This change allows setting a default MAKEOBJDIRPREFIX via local.sys.env.mk.

Sponsored by:	EMC / Isilon Storage Division
2016-01-07 00:19:03 +00:00
cperciva
7b9847ba48 Document that make's .POSIX: handling is broken. In fact, it has been
broken ever since it was added in November 1996.
2015-12-25 10:08:42 +00:00
br
e10e8212ee Start support for the RISC-V 64-bit architecture developed by UC Berkeley.
RISC-V is a new ISA designed to support computer research and education, and
is now become a standard open architecture for industry implementations.

This is a minimal set of changes required to run 'make kernel-toolchain'
using external (GNU) toolchain.

The FreeBSD/RISC-V project home: https://wiki.freebsd.org/riscv.

Reviewed by:	andrew, bdrewery, emaste, imp
Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
Differential Revision:	https://reviews.freebsd.org/D4445
2015-12-11 22:55:23 +00:00
araujo
8425221d14 Fix minor typos introduced on r292084.
Approved by:	rodrigc (mentor)
Differential Revision:	https://reviews.freebsd.org/D4495
2015-12-11 07:24:25 +00:00
imp
50899ae122 Move the inclusion of bsd.cpu.mk from sys.mk to bsd.opts.mk. However,
for historical behavior that ports depends on, include it if we're
inside the ports tree.

Differential Review: https://reviews.freebsd.org/D4383
Ports Exp run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=205021
2015-12-11 05:39:44 +00:00
bdrewery
52255645fb META MODE: Don't create .meta files when symlinking sources into the obj directory.
Tracking these leads to situations where meta mode will consider the
file to be out of date if /bin/sh or /bin/ln are newer than the source
file.  There's no reason for meta mode to do this as make is already
handling the rebuild dependency fine.

Sponsored by:	EMC / Isilon Storage Division
2015-11-25 19:44:43 +00:00
bdrewery
d1050e87e3 META MODE: Revert r287879 so that 'make -V' still uses AUTO_OBJ.
This has caused much confusion for myself as there are quite a lot of
variables that depend on having a proper ${.OBJDIR}.

Sponsored by:	EMC / Isilon Storage Division
2015-11-25 19:13:32 +00:00
sjg
6fee50ba24 Default MK_META_MODE from MK_DIRDEPS_BUILD
This allows most of the build to simply consider MK_META_MODE

Update to latest dirdeps.mk so we can do:

make -f dirdeps.mk bin/cat.i386

Reviewed by:	bdrewery
2015-11-16 22:50:23 +00:00
sjg
206f180a1a Rename META_MODE option to DIRDEPS_BUILD
This allows META_FILES option to be renamed META_MODE.
Also add META_COOKIE_TOUCH for use in targets that can benefit
from a cookie when in meta mode.

Differential Revision:	https://reviews.freebsd.org/D4153
Reviewed by:	bdrewery
2015-11-14 03:24:48 +00:00
bdrewery
11710a073f Add built-in ccache build support via WITH_CCACHE_BUILD option.
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build.  This is explained in
more detail upstream [1] [2].  It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times.  When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'.  Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports.  This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead.  ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss.  For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct.  Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it.  Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings.  GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual.  Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization.  There is also a memcached backend available [5].  Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout.  A
possible working layout is as follows:
  Source: /some/prefix/src1
  Source: /some/prefix/src2
  Source: /some/prefix/src3
  Objdir: /some/prefix/obj
  Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative.  Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files and ccache objects.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
  x buildworld-before
  + buildworld-ccache-empty
  * buildworld-ccache-full
  % buildworld-ccache-full-fastdep
  # buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |%            *                               #                                +|
  |%            *                               #                                +|
  |%            *                               #             xxx                +|
  |                                                           |A                  |
  |                                                                              A|
  |             A                                                                 |
  |A                                                                              |
  |                                             A                                 |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3          4519       4525.04       4520.73       4521.59     3.1104823
  Difference at 95.0% confidence
          758.027 +/- 43.4565
          20.1412% +/- 1.15466%
          (Student's t, pooled s = 19.1726)
  *   3       1823.08        1827.2       1825.62        1825.3     2.0785572
  Difference at 95.0% confidence
          -1938.26 +/- 43.298
          -51.5007% +/- 1.15045%
          (Student's t, pooled s = 19.1026)
  %   3       1266.96       1279.37       1270.47     1272.2667     6.3971113
  Difference at 95.0% confidence
          -2491.3 +/- 44.3704
          -66.1952% +/- 1.17895%
          (Student's t, pooled s = 19.5758)
  #   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-ccache-empty
  * buildkernel-ccache-empty-fastdep
  % buildkernel-ccache-full
  # buildkernel-ccache-full-fastdep
  @ buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |#                        @   %                  *                              |
  |#                        @   %                  *     x                      + |
  |#                        @   %                  *     xx                     ++|
  |                                                      MA                       |
  |                                                                             MA|
  |                                                A                              |
  |                             A                                                 |
  |A                                                                              |
  |                         A                                                     |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        727.97        731.91        728.06     729.31333     2.2492295
  Difference at 95.0% confidence
          156.88 +/- 4.17129
          27.4058% +/- 0.728695%
          (Student's t, pooled s = 1.84034)
  *   3         527.1        528.29        528.08     527.82333    0.63516402
  Difference at 95.0% confidence
          -44.61 +/- 2.33254
          -7.79305% +/- 0.407478%
          (Student's t, pooled s = 1.02909)
  %   3         400.4        401.05        400.62        400.69     0.3306055
  Difference at 95.0% confidence
          -171.743 +/- 2.16453
          -30.0023% +/- 0.378128%
          (Student's t, pooled s = 0.954969)
  #   3        201.94        203.34        202.28        202.52    0.73020545
  Difference at 95.0% confidence
          -369.913 +/- 2.40293
          -64.6212% +/- 0.419774%
          (Student's t, pooled s = 1.06015)
  @   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR:		182944 [4]
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
Relnotes:	yes
2015-11-08 00:50:18 +00:00
bdrewery
d506826439 Rework the 'make -n -n' feature such that '-n' recurses and '-N' does not.
Bmake has a documented feature of '-N' to skip executing commands which is
specifically intended for debugging top-level builds and not recursing into
sub-directories.  This matches the older 'make -n' behavior we added which made
'-n -n' the recursing target and '-n' a non-recursing target.

Removing the '-n -n' feature allows the build to work as documented in
the bmake manpage with '-n' and '-N'.  The older '-n -n' feature was also
not documented anywhere that I could see.

Note that the ${_+_} var is still needed as currently bmake incorrectly
executes '+' commands when '-N' is specified.

The '-n' and '-n -n' features were broken for several reasons prior to this.
r251748 made '_+_' never expand with '-n -n' which resulted in many
sub-directories not being visited until fixed 2 years later in r288391, and
many targets were given .MAKE over the past few years which resulted in
non-sub-make commands, such as rm and ln and mtree, to be executed.

This should also allow removing some indirection hacks in bsd.subdir.mk and
other cases of .USE that have a .MAKE by using '+'.

Sponsored by:	EMC / Isilon Storage Division
Discussed on:	arch@ (mostly silence)
2015-10-17 16:42:54 +00:00
bdrewery
af51acc847 Explicitly enable .MAKE.ALWAYS_PASS_JOB_QUEUE for bmake.
This is a NOP as r254419 enabled this by default in bmake.  Add it here though
to ensure it is known that we are using this as a default and in case a
bmake import removes the default we have.

This tells bmake to always pass job tokens into sub-commands.  Otherwise
it would only do so if the target being built depended on the special
.MAKE target (which causes _all_ commands to be executed with -n as well)
or if the command matches '${MAKE}/${.MAKE}/$(MAKE)/$(.MAKE)/make' (before
expansion, so ${LIB32WMAKE} would not qualify).  Using '+' on a command
(which runs the command with -n) would not pass the job token even though it
is a documented way to achieve the .MAKE effect on a command.

Sponsored by:	EMC / Isilon Storage Division
2015-09-25 22:51:30 +00:00
bdrewery
8e99ed192a META_MODE: Follow-up r287879 and have 'make -V .OBJDIR' still invoke auto.obj.mk.
When inspecting this value it is more expected to have it show the
automatically-created directory value rather than CURDIR.

Sponsored by:	EMC / Isilon Storage Division
2015-09-23 20:46:23 +00:00
bdrewery
a651b0b81d src.conf.5: Make it self-documenting that the mkopts are environment-only.
Sponsored by:	EMC / Isilon Storage Division
2015-09-17 20:33:52 +00:00
bdrewery
ae3bb164dd Include bsd.mkopt.mk after local.sys.env.mk (which includes /etc/src-env.conf).
This will allow setting WITH_META_MODE in that file rather that requiring
it to be set in the environment.

Sponsored by:	EMC / Isilon Storage Division
2015-09-17 17:35:06 +00:00
bdrewery
f09841e04b Fix makeman creating obj directories due to turning on WITH_AUTO_OBJ.
r284708 addressed this slightly but seems to have put the make(showconfig)
guard in the wrong place.  Rather than guard setting the default obj directory,
guard inclusion of auto.obj.mk.  This avoids creating SRCTOP/obj and
SRCTOP/release/obj when running makeman.

Sponsored by:	EMC / Isilon Storage Division
2015-09-17 16:38:32 +00:00
bdrewery
a9d2b4e7d3 META_MODE: Don't create obj directories automatically when running make -V.
Sponsored by:	EMC / Isilon Storage Division
2015-09-16 23:01:16 +00:00
imp
02dae40cb4 Make sys.mk more compatible with fmake by refraining from using :U
modifiers.

Differential Revision: https://reviews.freebsd.org/D3228
2015-08-27 02:59:48 +00:00
rpaulo
2c035421f7 Introduce a new make variable: NMFLAGS.
As the name indicates, these are flags to pass to nm(1).  The newer
binutils have a plugin mechanism so, to build something with LLVM's
LTO, we need to pass flags to nm(1).  This commit also extends
lorder(1) to pass NMFLAGS to nm(1).
2015-08-14 22:54:52 +00:00
sjg
9b22f67e32 Move include of make.conf back to its old position.
This means moving include of local.sys.mk and src.sys.mk too.
Introduce new includes to take the early slot, for the purpose
of being able to influence toolchains and the like.

Differential Revision:	D2860
Reviewed by:	imp
2015-06-19 14:56:24 +00:00
sjg
008d7c831f Add META_MODE support.
Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.

Still need to add real targets under targets/ to build packages.

Differential Revision:       D2796
Reviewed by: brooks imp
2015-06-13 19:20:56 +00:00
sjg
30006bb978 Building on fmake is no longer possible so removed tests for bmake. 2015-06-10 18:14:38 +00:00
sjg
a95f28426e Include bsd.mkopt.mk before local.sys.mk
Also use __DEFAULT_DEPENDENT_OPTIONS for options that
generally depend on META_MODE.
Deal with MK_META_MODE and MK_AUTO_OBJ directly.
Also allow MK_META_FILES if no -B
this is very handy for getting meta files from say buildworld
2015-05-26 21:51:36 +00:00
sjg
7566a01eaa Add options for auto obj and meta mode to sys.mk 2015-05-24 02:46:08 +00:00
markj
ee504eba70 Add a DTRACEFLAGS variable, which can be used to pass additional variables
to dtrace(1) invocations during a build. This change includes -C in the
default flags, which has dtrace(1) run input scripts through the
preprocessor. While here, sort the definitions of CP and CPP in sys.mk.

Differential Revision:	https://reviews.freebsd.org/D2204
Reviewed by:		imp, rpaulo (previous revision)
2015-04-08 02:43:05 +00:00
dim
d202cb2c64 Allow relative pathnames in SRCS, so as to enable building software
which includes more than one file with the same name, in different
directories.

For example, setting:

SRCS+=	foo/foo.c bar/foo.c baz/foo.c

will now create separate objdirs 'foo', 'bar' and 'baz' for each of the
sources in the list, and use those objdirs for the corresponding object
files.

Reviewed by:	brooks, imp
MFC after:	1 week
Differential Revision: https://reviews.freebsd.org/D1984
2015-03-14 12:29:44 +00:00
will
afad9375d8 Add a ${CP} alias for copying files in the build.
Some users build FreeBSD as non-root in Perforce workspaces.  By default,
Perforce sets files read-only unless they're explicitly being edited.
As a result, the -f argument must be used to cp in order to override the
read-only flag when copying source files to object directories.  Bare use of
'cp' should be avoided in the future.

Update all current users of 'cp' in the src tree.

Reviewed by:	emaste
MFC after:	1 week
Sponsored by:	Spectra Logic
2015-01-16 21:39:08 +00:00
sjg
b137080f19 Merge from head@274682 2014-11-19 01:07:58 +00:00
andrew
5a1bee9f60 Add OBJDUMP to sys.mk. The EFI loader uses it however without this it can
be set but empty.

Obtained from:	ABT Systems
Sponsored by:	The FreeBSD Foundation
2014-11-14 14:45:36 +00:00
bapt
82f995d6f3 Make sure SIZE is defined early 2014-10-10 06:09:47 +00:00
sjg
d7cd1d425c Merge head from 7/28 2014-08-19 06:50:54 +00:00
sjg
f96bc8e824 Reviewed by: imp
LDFLAGS is supposed to be given to CC not LD.
Define _LDFLAGS as a filtered version of LDFLAGS safe to give to LD
2014-07-26 04:38:09 +00:00
imp
578932e7e1 Move inclusion of /etc/make.conf and others to old location (this also
moves local.sys.mk). The new location broke adding things to CXXFLAGS
in /etc/src.conf with +=. Move it back until that's sorted out...
2014-05-17 20:31:40 +00:00
imp
3d089b2d54 Read in SRCCONF early and consistently, if src.sys.mk exists, which is
should for all normal builds. Read /etc/make.conf earlier than before,
but consistently before SRCCONF and local.sys.mk.
2014-05-17 01:47:06 +00:00
imp
6b50de24a4 Remove some useless, commented out code. Remove name space polution in
the POSIX case by moving more things under !Posix part of an if.
2014-05-10 16:38:32 +00:00
imp
6d25f9c86f Don't apply ctf conversions in POSIX mode. These can't happen there
because they pollute the POSIX environment, which doens't allow
for these extentions. ctf conversions are really only relevant when
used in coordination with the rest of the bsd*.mk system anyway.
Leave them in place for the normal, non-posix enviornment since
they are quite useful there.
2014-04-13 05:21:48 +00:00
des
51c7df1f50 Introduce RANLIBFLAGS to mirror ARFLAGS and add -D to both. This sets
all timestamps in static libraries to 0 so that consecutive builds
from the same source, even on different machines, produce identical
libraries.

MFC after:	3 weeks
2014-04-12 12:44:09 +00:00
andrew
8842d57879 Add a new ARM TARGET_ARCH, armv6hf. This is considered experimental.
This targets the existing ARMv6 and ARMv7 SoCs that contain a VFP unit.
This is an optional coprocessors may not be present in all devices, however
it appears to be in all current SoCs we support.

armv6hf targets the VFP variant of the ARM EABI and our copy of gcc is too
old to support this. Because of this there are a number of WITH/WITHOUT
options that are unsupported and must be left as the default value. The
options and their required value are:
 * WITH_ARM_EABI
 * WITHOUT_GCC
 * WITHOUT_GNUCXX

In addition, without an external toolchain, the following need to be left
as their default:
 * WITH_CLANG
 * WITH_CLANG_IS_CC

As there is a different method of passing float and double values to
functions the ABI is incompatible with existing armv6 binaries. To use
this a full rebuild of world is required. Because no floating point values
are passed into the kernel an armv6 kernel with VFP enabled will work with
an armv6hf userland and vice versa.
2014-03-23 12:49:25 +00:00
sjg
142ae77478 Use .SHELL to tell bmake to use 'set -e' when running scripts
since most FreeBSD makefiles it is in effect.

Move the other bmake compatability knobs out of the POSIX block.

Reviewed by: obrien
2013-08-27 23:09:34 +00:00
eadler
afde4801cd Remove variable added by ru@ in r94940 that is no longer used by source or ports in share/mk/sys.mk
PR:		conf/155737
Reviewed by:	ed (two years ago!)
Exp-Run by:	bdrewery
2013-06-23 20:34:54 +00:00
sjg
b66c288337 We cannot remove the _+_ trick, until old make is completely deprecated.
But we don't want to set it to + for bmake since it breaks make -N
which is used to supress the normal handling of targets marked with .MAKE
(which seems broken in fmake and might be why _+_ was introduced).
Add some comments to explain what's gong on.

Reviewed by:	obrien
2013-06-14 16:24:07 +00:00
sjg
41b9477759 Override bmake's default MAKEFILE_PREFERENCE 2013-06-07 19:45:04 +00:00
sjg
6702b6086c Tell bmake to use the FreeBSD preferred makefile preference list.
PR:		177593
Reviewed by:	obrien
2013-04-03 16:20:21 +00:00
des
caf94f2cca Remove all support for legacy NOFOO and NO_FOO build options. 2013-01-10 22:44:19 +00:00