Commit Graph

1720 Commits

Author SHA1 Message Date
Bryan Drewery
b7c6a063fe Only build libzfs_core in 'make libraries' if needed.
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-07-25 20:43:37 +00:00
Bryan Drewery
63c534e716 Remove unneeded dependency for libzfs.
This dependency does nothing since cddl/lib/libzfs is never
added into the 'make libraries' dependency chain
directly.

MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-07-25 20:43:26 +00:00
Bryan Drewery
1ae1311d9c NO_CLEAN: Utilize delete-old to remove old orphaned libraries/headers in WORLDTMP.
This prevents situations with -DNO_CLEAN from finding stale headers or
libraries in places that no longer exist or have moved.  It avoids
the need to remove all of WORLDTMP by reusing what we already know
is obsolete.

MFC after:	1 month
Sponsored by:	Dell EMC Isilon
2017-07-24 23:32:46 +00:00
Bryan Drewery
dc75d44eca The .depend.obj cleanup hacks are only needed with -DNO_CLEAN.
Sponsored by:	Dell EMC Isilon
2017-07-24 23:32:43 +00:00
Bryan Drewery
cd2831005b Move llvm Options.inc hack from r321433 for NO_CLEAN to lib/clang/libllvm.
The files are only ever generated to .OBJDIR, not to WORLDTMP (as a
sysroot) and are only ever included from a compilation.  So using
a beforebuild target here removes the file before the compilation
tries to include it.

MFC after:	2 months
X-MFC-With:	r321369
2017-07-24 23:32:24 +00:00
Dimitry Andric
edcc769f3f Cleanup stale Options.inc files from the previous libllvm build for
clang 4.0.0.  Otherwise, these can get included before the two newly
generated ones (which are different) for clang 5.0.0.

Reported by:	Mark Millard
MFC after:	2 months
X-MFC-With:	r321369
2017-07-24 18:52:40 +00:00
Bryan Drewery
0f827d7a96 makeman: Don't show META_MODE ABI rebuild warnings.
Reported by:	dim
Sponsored by:	Dell EMC Isilon
2017-07-08 17:53:00 +00:00
Bryan Drewery
9bb3819b32 SYSTEM_COMPILER: Ensure there is not a stale compiler in WORLDTMP.
In a scenario of cross-building it is possible that an OBJDIR's WORLDTMP
contains an older compiler in WORLDTMP/usr/bin/cc that is not rebuilt
if SYSTEM_COMPILER logic is triggered.  This compiler was still
incorrectly used.  Address this by removing WORLDTMP/usr/bin/cc and all
of the hardlinked files associated with it.  Also do this for c++ for
GCC builds.

Sponsored by:	Dell EMC Isilon
MFC after:	1 week
2017-07-08 16:39:55 +00:00
Bryan Drewery
f45c780573 Fix create-kernel-packages with multiple BUILDKERNELS after r320284
Submitted by:	Kyle Evans <kevans91@ksu.edu>
Reviewed by:	ngie
2017-07-05 19:24:38 +00:00
Ed Maste
a28412b0db Clean up stale dependencies after r320278
Our current approach to dependency tracking cannot cope with switching
generated asm syscall stubs into C wrappers. Perpetuate the hack in
Makefile.inc1 to paper over the problem until we can take a holistic
approach to fixing dependency problems.

Differential Revision:	https://reviews.freebsd.org/D11344
2017-06-24 18:37:51 +00:00
Bryan Drewery
0369270f71 NO_ROOT: Remove excessive // when DESTDIR/DISTDIR are empty.
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-06-23 19:03:31 +00:00
Bryan Drewery
2d5cc53891 compiler-metadata: Properly handle cross-build OBJDIR.
MFC after:	3 days
Sponsored by:	Dell EMC Isilon
2017-06-23 18:27:03 +00:00
Bryan Drewery
275219cfe7 Expose only the create-packages-* targets since they set needed DEST/DIRDIR.
The other targets just fail confusingly otherwise.

MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-06-23 18:27:00 +00:00
Bryan Drewery
10c935294b packages: Parallelize individual kernel packaging.
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-06-23 18:26:57 +00:00
Bryan Drewery
f77b03aa2b packages: Allow actually building individual world packages in parallel.
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-06-23 18:26:54 +00:00
Bryan Drewery
48c22674a8 packages: Allow creating kernel/world packages in parallel.
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-06-23 18:26:51 +00:00
Bryan Drewery
00c44825d6 packages: Allow staging world/kernel in parallel.
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-06-23 18:26:47 +00:00
Bryan Drewery
dd7b4c153a packages: Allow stageworld/stagekernel to run with make jobs.
The -B was originally added in projects/release-pkg r289381 as a copy
of what 'make world' did at the time.  The -B was removed from
the 'installworld' call in 'world' in r303844 though.  The staging
of files is safe to run in parallel.

MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-06-23 18:26:33 +00:00
Bryan Drewery
d20bd9e42b Set compiler metadata for stageworld/distributeworld.
This fixes LD errors during 'make packages' but also for the unlikely case of
'buildworld' on 1 system and 'packages' on another [1].

PR:		212877 [1]
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-06-23 16:38:46 +00:00
Bryan Drewery
81b4423f05 Pass along LINKER_* vars during installworld and show in test-system-compiler.
Sponsored by:	Dell EMC Isilon
2017-06-22 21:03:30 +00:00
Bryan Drewery
817366f1c1 buildworld: Pass which world phase the build is in down to submakes.
This is useful for having directories behave differently depending
on the phase - such as enabling SUBDIR_PARALLEL or disabling
redundant building of library directories already done by
earlier 'make _libraries'.

Sponsored by:	Dell EMC Isilon
2017-06-20 22:08:02 +00:00
Bryan Drewery
35fb812adf buildworld: Define SYSROOT to WORLDTMP.
This is to allow downstream Makefiles to know for sure they are building
against a sysroot rather than only depending on ${DESTDIR} or other
assumptions.

This also exports it into buildenv.

MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-06-19 18:08:20 +00:00
Ed Maste
ff6c17c94e remove stale dependencies for utimens* wrappers removed in r319663
Use a similar approach to r318957 (which was for ptrace dependencies):
grep the .depend file for the old source file and delete the stale
dependency if found.

Reviewed by:	bdrewery
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D11091
2017-06-12 19:45:05 +00:00
Baptiste Daroussin
69db934679 Fix src.conf.5 generation
When makeman is generating src.conf(5) it tries to test all variation of options
including WITH_DIRDEPS_BUILD. it results in an error when filemon(4) is not
loaded.

Export variables that are needed to prevent this behaviour.

Helped by:	sjg
2017-06-08 00:41:25 +00:00
Bryan Drewery
c8fab231f1 META_MODE: Force a host tool rebuild for ino64 in r318736.
Sponsored by:	Dell EMC Isilon
2017-06-05 05:17:42 +00:00
Bryan Drewery
1a9cc831dd META_MODE: Add framework to force rebuilding for major ABI changes.
Normally META_MODE ignores host files for "meta mode" decisions on whether a
file should be rebuilt or not.  This is because a simple installworld can
update timestamps and cause the next build to rebuild all host tools, when the
previous ones may not have any changes in the source tree.  These tools are
normally still ABI compatible.  They are only rebuilt if NO_META_IGNORE_HOST is
set from the workaround/hack in r301467.

One of the major problems with this is when a host tool has objects spread
across many revisions that have mixed-ABI.  For example, if struct stat were to
change on the host, some objects for a tool may have different ideas of that
struct's definition.  If just 1 source file were modified and rebuilt and
linked into the tool, then that toll will have mixed-ABI objects and crash.
This exact thing happened with the ino64 commit in r301467 followed by a
trivial update to libbfd in r318750.  The resulting binary would crash in
buildworld.

Sponsored by:	Dell EMC Isilon
2017-06-05 05:17:39 +00:00
Bryan Drewery
86f2d54ff0 native-xtools: Connect makewhatis.
This speeds up some port staging, namely perl.
2017-06-03 23:13:46 +00:00
Bryan Drewery
9f68573426 native-xtools: Connect lld.
This will ensure that aarch64 gets a working native /usr/bin/ld rather
than requiring the aarch64-binutils hack in Poudriere, or emulating
the aarch64 lld.

PR:		217189
Reported by:	swills, jbeich
2017-06-02 00:57:59 +00:00
Ed Maste
bd1c3bad76 add a rescue/sh sanity check before installworld on the running system
FreeBSD does not guarantee kernel forward compatibility (that is,
running a newer userland on an older kernel).  The documented upgrade
procedure specifies that installkernel should be performed, followed by
a reboot and then installworld.  As a sanity check when installing onto
the running system (DESTDIR is / or unset), attempt to run "sh echo OK"
using rescue from the objdir.  If rescue fails (e.g., because the system
has not been rebooted and the "old" kernel lacks a system call required
by the to-be-installed world), abort the installation.

This should avoid ino64 foot-shooting when the proper upgrade procedure
is not followed.

Reviewed by:	allanjude, gjb, kib
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D10987
2017-05-30 15:51:48 +00:00
Ed Maste
55b87d4621 rm stale ptrace dependencies after r305012
This is similar to r318912, except that ptrace.[sS] was previously a
file in the source tree, not a generated assembly wrapper.

Check for the existence of ptrace.[sS] in the .depend file to determine
if we have to clean it up.  This is a bit hackish and will not be left
in place indefinitely, but provides a useful example case when
investigating a better solution in bmake.

Reviewed by:	bdrewery
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D10930
2017-05-26 16:03:28 +00:00
Ed Maste
90d4415d20 libc: rm stale generated files which are no longer syscalls
This is an attempt to help -DNO_CLEAN builds after r302092 (which
removed the pipe libc syscall wrapper) and r318736 (which removed
getdents, lstat, mknod, and stat).

Dependencies cannot cope with certain source tree changes,
particularly with respect to removing source files and replacing
generated files.  Handle these cases from _worldtmp in an ad-hoc
fashion.

Reviewed by:	bdrewery, cem
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D10876
2017-05-26 00:51:05 +00:00
Baptiste Daroussin
ac4ef3f9a9 Now that roff documentation has been disconnected from the build, it is no
longer necessary to have groff(1) as a bootstrap tool
2017-05-25 16:31:53 +00:00
Bryan Drewery
c596c2876f Add a -DNO_LIBS to skip building the libraries phase as well.
This is useful for cases where -DWORLDFAST is useful.

Sponsored by:	Dell EMC Isilon
2017-05-09 23:58:07 +00:00
Bryan Drewery
16892c7713 Support -DWORLDFAST to skip all build steps up to 'libraries' and 'everything'.
This allows for building the world against the already-created
host/sysroot environment.  It is not overly useful outside of cases of
large-impact changes such as a testing a new compiler.  It will
allow quickly getting back to an error in the target-phases of the
build where a new compiler is being used.

Sponsored by:	Dell EMC Isilon
2017-05-09 20:21:58 +00:00
Bryan Drewery
6d75a7a852 Support skipping 'make obj' tree-walk.
This is part of a larger effort for WITH_AUTO_OBJ and a WORLDFAST
option.

Sponsored by:	Dell EMC Isilon
2017-05-09 20:21:38 +00:00
John Baldwin
d5392a4191 Honor WITHOUT_LIB32 on mips64.
The closing paren for the list of architectures that should enable LIB32
by default was in the wrong place resulting in LIB32 always be enabled on
mips64 regardless of WITH_LIB32/WITHOUT_LIB32.

Submitted by:	Alex Richardson <Alexander.Richardson@cl.cam.ac.uk>
Obtained from:	CheriBSD
Sponsored by:	DARPA / AFRL
2017-05-08 18:51:13 +00:00
Baptiste Daroussin
a17665bcef Replace again GNU diff with BSD diff
After a firts failed attempt, BSD diff is now good enough to replace
GNU diff.

Relnotes:	yes
2017-04-20 19:24:51 +00:00
Glen Barber
5e50a31863 Use relative symlink for 'latest' to allow repo to work after being
moved.

Submitted by:	woodsb02
MFC after:	3 days
PR:		217197
Sponsored by:	The FreeBSD Foundation
2017-04-18 19:44:34 +00:00
Bryan Drewery
848d5e929b Cache compiler metadata and reuse it at installworld time.
Right after cross-tools, a compiler-metadata.mk file is created that
stores all of the bsd.compiler.mk metadata.  It is then read in
with a fail-safe during installworld time.

The file is explicitly removed when invoking cross-tools to ensure that
a stale file is not left around from odd manual 'make _cross-tools' ->
'make installworld' invocations.

This fixes several issues:
    - With WITH_SYSTEM_COMPILER (default yes on head and no on releng/11.0):
      If you build on a system where the bootstrap compiler does not
      build due to the host compiler matching the in-tree one, but then
      installworld on another system where that logic fails (a
      bootstrap compiler is needed), the installworld immediately fails
      with:
           sh: cc: not found
      Note that fixing this logic may then hit a case where a rebuild is
      attempted in installworld.  Normally cc would be ran with
      'CFLAGS+=ERROR-tried-to-rebuild-during-make-install' to cause an
      error such as:
          cc: error: no such file or directory: 'ERROR-tried-to-rebuild-during-make-install'
      However, now it will just fail with the 'cc: not found' error.
      Inspection of the compile line will show
      'ERROR-tried-to-rebuild-during-make-install';  It's not useful to
      set CC to anything other than 'cc' during install as it is more
      helpful to see the attempted compile rather than some other bogus
      error.
    - This now avoids running bsd.compiler.mk (cc executions) even more
      during installworld.  There are compiler-dependent SUBDIR in the
      tree which required having a compiler during install.

There is at least 1 case where CC is still executed in the install,
such as from a LOOKUP!= in secure/lib/libcrypto/Makefile.inc checking
for 'vzeroall' support.  This is not significant for installworld
as the lookup has a fallback (and hides its error) and only modifies CFLAGS,
thus it's not worth fixing.

PR:		212877
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-04-13 22:07:34 +00:00
Bryan Drewery
d5b29b60f8 Pass COMPILER_FEATURES down to submakes for installworld.
This is for WITH_SYSTEM_COMPILER, WITHOUT_CROSS_COMPILER, external
compiler, etc.

MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-04-13 19:27:46 +00:00
Ed Maste
ffe63c8bf0 Introduce LLD_BOOTSTRAP to control lld as bootstrap linker
Add WITH_LLD_BOOTSTRAP and WITHOUT_LLD_BOOTSTRAP knobs, similar to the
Clang bootstrap knobs.

Reviewed by:	dim
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D10249
2017-04-09 01:35:19 +00:00
Ed Maste
1bf5e133cb do not require binutils port when using lld as ld
r279908 added logic to Makefile.inc1 to automatically set
CROSS_BINUTILS_PREFIX for architectures not supported by the in-tree
binutils: arm64 when first introduced, and later riscv64 as well.

LLVM's LLD linker is now included in the base system, and is enabled by
default for arm64 and capable of linking world and kernel. Thus, avoid
automatically setting CROSS_BINUTILS_PREFIX and requiring the binutils
port if WITH_LLD_IS_LD is true.

Reviewed by:	kan
Relnotes:	Yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D10310
2017-04-07 23:37:22 +00:00
Bryan Drewery
dbbb03c244 META_MODE: Fix build-tools still sometimes rebuilding during target build.
In a cross-build, the build-tools are native host binaries.  We do not
want to rebuild them when building for the target.  Bmake previously
did not support checking .NOMETA on an existing target, so .NOMETA_CMP
was used here.  However, .NOMETA_CMP still triggers meta mode conditions
if the number of commands or the command changes.  In r312467 the paths
to build ncurses files were modified and thus triggered meta mode to
rebuild the build tools (make_keys, make_hash) in ncurses during the
target build.  Bmake 20160604 committed in r301462 changed .NOMETA to
also skip meta mode logic for an existing .meta file as well, thus it
is now the proper fix here.

I explored moving the build-tools output to WORLDTMP/tools with
relatively good success, but have concerns that doing so would be
problematic for downstream vendors who use LOCAL_TOOL_DIRS and
expect the tools to be in current OBJDIR for the target.  It also
adds more complexity into finding the tools during target build
and handling of where they are for rescue/rescue and
mkcsmapper_static/mkesdb_static which should really not be connected in
build-tools anyway.

MFC after:	2 weeks
Reported by:	many
Sponsored by:	Dell EMC Isilon
2017-04-06 18:21:59 +00:00
Bryan Drewery
3507006763 Rename _cc to _gcc to be more clear.
MFC after:	1 week
Sponsored by:	Dell EMC Isilon
2017-03-21 22:09:00 +00:00
Baptiste Daroussin
2c5ddcf8b6 Temporary readd GNU diff
etcupdate requires --change-group-format it is not easy to implement
in bsd diff so for now readd GNU diff
2017-03-19 17:19:59 +00:00
Baptiste Daroussin
5714b7010f Fix native-xtools after replacement of GNU diff 2017-03-12 03:33:20 +00:00
Baptiste Daroussin
97ab006d51 Remove the WITHOUT_MANDOCDB option
mandoc database is activated since FreeBSD 11.0, let's remove the previous
database format for FreeBSD 12.0
2017-03-11 06:51:21 +00:00
Bryan Drewery
0e4e8f1b33 Added comments for why nmtree/libmd are bootstrapped. 2017-03-06 20:32:17 +00:00
Bryan Drewery
51c6e78d91 Fix bootstrapping mtree after r313404 for older systems.
r313404 made libnetbsd require sha384.h from libmd.  Libmd added it in
r292782.  Update BOOTSTRAPPING to account for this.

Reported by:	bde
Reviewed by:	ngie
2017-03-05 21:16:50 +00:00
Dimitry Andric
6ae9acde63 Merge ^/head r313896 through r314128. 2017-02-23 07:45:58 +00:00
Warner Losh
6fc94bc49f Document why cat is a bootstrap tool. 2017-02-21 18:49:30 +00:00
Enji Cooper
74464d3ca4 Include ${LOCALBASE}/bin in $PATH when running "make checkworld"
Some of the tests in devel/atf // devel/kyua rely on the tools being in $PATH,
which means that the tests fail when run via "make checkworld" because $PATH
is restricted to exclude directory elements like "${LOCALBASE}/bin".

MFC after:	1 week
Sponsored by:	Dell EMC Isilon
2017-02-18 21:47:32 +00:00
Enji Cooper
66df1425ff Quote path to doxygen/kyua in test(1) -x check
This is a basic stopgap against ${LOCALBASE} containing spaces in it

MFC after:	1 week
Sponsored by:	Dell EMC Isilon
2017-02-18 21:41:50 +00:00
Bryan Drewery
68a558eca2 xdev: Fix after libc++ update in r300873.
The xdev build needed the same fixes as libcompat and external toolchain
support needed for handling of --sysroot, -L, -B, etc.

MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-02-18 00:08:13 +00:00
Bryan Drewery
e36b831f4b xdev: Build yacc which is needed for recent libpcap updates.
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-02-17 22:51:34 +00:00
Bryan Drewery
9e580940c1 META_MODE+xdev: Don't rebuild build-tools during normal build.
See r297997 for more information.

MFC after:	1 week
Sponsored by:	Dell EMC Isilon
2017-02-17 22:51:21 +00:00
Dimitry Andric
f9edb08480 Merge ^/head r313055 through r313300. 2017-02-05 20:03:05 +00:00
Bryan Drewery
e297227038 Remove LOCAL_LIB_DIRS warning added in r275839.
The case for which this was added, r274807, causes this warning to
always show.  LOCAL_DIRS=foo LOCAL_LIB_DIRS=foo/lib.  The only case in
which r274807 is a problem is if foo/Makefile does not contain
SUBDIR+=lib, which is a normal convention.  LOCAL_LIB_DIRS is a special
hack only to get a library into the _generic_libs list for the
'make libraries' bootstrapping phase.  The old behavior changed in
r274807 was only in head during the 10.0 cycle, so the warning was
only ever needed until release anyhow.

Reported by:	ngie
MFC after:	1 week
Sponsored by:	Dell EMC Isilon
2017-02-04 02:15:49 +00:00
Enji Cooper
c6dc13326b Fix typo in variable name (_REDUNDENT_LIB_DIRS -> _REDUNDANT_LIB_DIRS)
MFC after:	3 days
Sponsored by:	Dell EMC Isilon
2017-02-04 01:21:48 +00:00
Bryan Drewery
27f134b64a native-xtools: Add missing readelf.
The switch to elftoolchain's readelf in r280859 caused native-xtools
to no longer build readelf.  This fixes poudriere builds not using
a native readelf when expected.

Reported by:	strejda on freenode
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2017-02-03 16:27:23 +00:00
Dimitry Andric
65575c1424 Merge ^/head r312894 through r312967. 2017-01-29 22:00:47 +00:00
Yoshihiro Takahashi
2b375b4edd Remove pc98 support completely.
I thank all developers and contributors for pc98.

Relnotes:	yes
2017-01-28 02:22:15 +00:00
Ed Maste
d63b10f090 Update clang400 branch for LD_AS_LLD -> LD_IS_LLD 2017-01-27 20:49:43 +00:00
Dimitry Andric
14f850f3df Merge ^/head r312720 through r312893. 2017-01-27 20:01:46 +00:00
Ed Maste
f52e4bdfd7 Rename LLD_AS_LD to LLD_IS_LD, for consistency with CLANG_IS_CC
Reported by:	Dan McGregor <dan.mcgregor usask.ca>
2017-01-27 01:59:12 +00:00
Ed Maste
737872e978 Also apply WITH_LLD_AS_LD to build tools
Previously WITH_LLD_AS_LD installed LLD as /usr/bin/ld in the target
system, but still used the GNU BFD ld to link the binaries in that
target. LLD 4.0.0 can link the FreeBSD/amd64 world and kernel so use
LLD as the build-time linker as well when the knob is set.

Reviewed by:	dim
Relnotes:	Yes
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D9226
2017-01-25 21:05:48 +00:00
Baptiste Daroussin
7666f5006c Import mandoc cvs snapshot 20170121 (pre 1.14)
Note that mandoc does not use anymore sqlite3 but a home made database format
An important improvement has been made as well in makewhatis performance:
Tests on my laptop shows makewhatis on the entire system goes from 26s to 12s
2017-01-21 13:17:25 +00:00
John Baldwin
69b4d461be Enable /usr/lib32 for o32 binaries on mips64.
Build and install an o32 set of libraries on mips64 suitable for
running o32 binaries via COMPAT_FREEBSD32. Enable COMPAT_FREEBSD32 in
MALTA64.

Reviewed by:	jmallett, imp
Sponsored by:	DARPA / AFRL
Differential Revision:	https://reviews.freebsd.org/D9032
2017-01-06 23:30:54 +00:00
Enji Cooper
94ef145e6b Only bake krb5_config.h support in to ssh(3), etc if both MK_GSSAPI and
MK_KERBEROS_SUPPORT != no

This fixes the odd case where someone specified MK_GSSAPI=no and
MK_KERBEROS_SUPPORT=yes (which admittedly, probably doesn't make sense,
but the build system doesn't prevent this case today, and it didn't when
I filed the bug back in 2011 either).

MFC after:	2 weeks
PR:		159745
2017-01-02 20:29:50 +00:00
Dimitry Andric
715652a404 Make native-xtools build correctly after clang/llvm 3.9.0 import
During the clang/llvm 3.9.0 import, the build structure for it was
completely revamped.  This broke the native-xtools target.

It first attempts to build libllvmminimal, then the llvm-tblgen and
clang-tblgen executables, but these fail to link because they are linked
to the 'full' libllvm by default, as they normally are during the
'world' stage.

To make these link against libllvmminimal instead, define TOOLS_PREFIX,
similarly as during the bootstrap-tools phase.  The value itself is
empty, as we don't really want to use a prefix.

Reviewed by:	imp
PR:		215684
MFC after:	3 days
Differential Revision:	https://reviews.freebsd.org/D9026
2017-01-02 19:33:22 +00:00
Matthew Seaman
aeb76c0bb7 Revert r309339, thus re-instating r309314
The original problem with conflicting definitions of ${PKG_CMD} was
solved by r427523 in ports (see https://reviews.freebsd.org/D8677), so
this should be safe now.

Reviewed by:	gjb
Approved by:	gjb
Differential Revision:	https://reviews.freebsd.org/D8120
2016-12-16 23:05:47 +00:00
Baptiste Daroussin
629582ca46 Now that external gcc directly natively links to libc++ we can remove
the dirty hack made to fake libstdc++
2016-12-10 18:29:39 +00:00
Warner Losh
158c18ffb4 dd is currently a bootstrap tool. It really doesn't have any business
being a bootstrap tool. However, for reproducible build output,
FreeBSD added dd status=none because it was otherwise difficult to
suppress the status information, but retain any errors that might
happen. There's no real reason that dd has to be a build tool, other
than we use status=none unconditional. Remove dd from a bootstrap tool
entirely by only using status=none when available. This may also help
efforts to build the system on non-FreeBSD hosts as well.

Differential Revision: https://reviews.freebsd.org/D8605
2016-12-02 14:44:38 +00:00
Glen Barber
ebc7f2b8de Revert r309314, which breaks installing ports.
Requested by:	antoine
Differential Revision:	https://reviews.freebsd.org/D8120 (related)
Sponsored by:	The FreeBSD Foundation
2016-11-30 22:00:25 +00:00
Dimitry Andric
26e287836b Cleanup old debug dirs in delete-old-dirs target
Any .debug or .symbols files under /usr/lib/debug which correspond to
OLD_FILES entries in ObsoleteFiles.inc are also automatically cleaned up
by the delete-old target.  Make this also apply to any OLD_DIRS entries.

Reviewed by:	emaste
MFC after:	3 days
Differential Revision: https://reviews.freebsd.org/D8683
2016-11-30 19:06:47 +00:00
Matthew Seaman
91c915facf Allow a user-overridable setting 'PKG_CMD' to control the command used
to create a repo during 'make packages'

This would have been useful for a situation I found myself in where
pkg(8) had been upgraded to a version that wanted the FBSD_1.5 ABI
version but libc.so.7 had not been upgraded, and only provided
FBSD_1.4. I found I needed to update libc in order to run pkg, and I
also needed to use pkg to update libc... Which is why pkg-static
exists, but there's currently no way to tell the build system to use
pkg-static instead of pkg.

This creates a variable PKG_CMD, default value 'pkg', that can be
overridden from the command line.

Reviewed by:	gjb
Approved by:	gjb
Differential Revision:	https://reviews.freebsd.org/D8120
2016-11-30 07:16:29 +00:00
Ed Maste
0aa5466e7d Add WITH_LLD_AS_LD build knob
If set it installs LLD as /usr/bin/ld.  LLD (as of version 3.9) is not
capable of linking the world and kernel, but can self-host and link many
substantial applications. GNU ld continues to be used for the world and
kernel build, regardless of how this knob is set.

It is on by default for arm64, and off for all other CPU architectures.

Sponsored by:	The FreeBSD Foundation
2016-11-25 13:15:28 +00:00
Dimitry Andric
67bc8c8b9e Merge ^/head r308491 through r308841. 2016-11-19 16:05:55 +00:00
Ruslan Bukin
7804dd5212 Add full softfloat and hardfloat support for RISC-V.
Hardfloat is now default (use riscv64sf as TARGET_ARCH
for softfloat).

Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D8529
2016-11-16 15:21:32 +00:00
Bryan Drewery
7c6d0bb1a0 Move libsysdecode-specific hack out of buildworld.
This should fix the lib32 build since it was not removing the generated
ioctl.c.  This file is generated by a find(1) call, so cannot use normal
dependency tracking methods.

Reported by:	jhb
MFC after:	2 weeks
Sponsored by:	Dell EMC Isilon
2016-11-13 00:11:15 +00:00
Dimitry Andric
2828dafcf3 Merge ^/head r308227 through r308490. 2016-11-10 22:12:19 +00:00
Antoine Brodin
111bf579e3 Add limits(1) to native-xtools so that it can be used in qemu user-mode jails 2016-11-10 16:27:34 +00:00
Ed Maste
c7fbd7722d Connect new LLVM-based libgcc_eh & libgcc_s to the build
Compiler-rt and LLVM's libunwind provide a suitable replacement for
libgcc.a, libgcc_eh.a, and libgcc_s.so.

Remove the now-unused LLVM_LIBUNWIND block from gnu/lib/libgcc.

PR:		213480 [exp-run]
Reviewed by:	brooks, ed
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D8189
2016-11-04 19:35:49 +00:00
Dimitry Andric
a2b802ce70 Merge ^/head r303250 through r308226. 2016-11-02 19:18:24 +00:00
Jonathan Anderson
e4195e2e12 Add rules to build LLVM IR binaries and libraries.
Running `make libfoo.ll` or `make libfoo.bc` within a library directory
will now give us an LLVM IR version of the library, and `make foo.full.ll`
or `make foo.full.bc` will give us an IR version of a binary.

As part of this change, we add an LLVM_LINK variable to sys.mk that can be
specified/overridden using an external toolchain.

Reviewed by:	bdrewery, brooks
Approved by:	rwatson (mentor)
Sponsored by:	DARPA, AFRL
Differential Revision:	https://reviews.freebsd.org/D8388
2016-11-01 21:27:42 +00:00
Dimitry Andric
02ebdc7823 Merge ^/head r307736 through r308146. 2016-10-31 19:02:42 +00:00
Ruslan Bukin
5bca221511 Add full softfloat and hardfloat support for MIPS.
This adds new target architectures for hardfloat:
mipselhf mipshf mips64elhf mips64hf.

Tested in QEMU only.

Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
Differential Revision:	https://reviews.freebsd.org/D8376
2016-10-31 15:33:58 +00:00
Warner Losh
9013696194 Tweak the UPDATING message a bit about the upgrade path.
Add some automation into Makefile.inc1 to to enforce known good
upgrade from source paths.
2016-10-23 18:00:08 +00:00
Justin Hibbits
dc9b124d66 Create a new MACHINE_ARCH for Freescale PowerPC e500v2
Summary:
The Freescale e500v2 PowerPC core does not use a standard FPU.
Instead, it uses a Signal Processing Engine (SPE)--a DSP-style vector processor
unit, which doubles as a FPU.  The PowerPC SPE ABI is incompatible with the
stock powerpc ABI, so a new MACHINE_ARCH was created to deal with this.
Additionaly, the SPE opcodes overlap with Altivec, so these are mutually
exclusive.  Taking advantage of this fact, a new file, powerpc/booke/spe.c, was
created with the same function set as in powerpc/powerpc/altivec.c, so it
becomes effectively a drop-in replacement.  setjmp/longjmp were modified to save
the upper 32-bits of the now-64-bit GPRs (upper 32-bits are only accessible by
the SPE).

Note: This does _not_ support the SPE in the e500v1, as the e500v1 SPE does not
support double-precision floating point.

Also, without a new MACHINE_ARCH it would be impossible to provide binary
packages which utilize the SPE.

Additionally, no work has been done to support ports, work is needed for this.
This also means no newer gcc can yet be used.  However, gcc's powerpc support
has been refactored which would make adding a powerpcspe-freebsd target very
easy.

Test Plan:
This was lightly tested on a RouterBoard RB800 and an AmigaOne A1222
(P1022-based) board, compiled against the new ABI.  Base system utilities
(/bin/sh, /bin/ls, etc) still function appropriately, the system is able to boot
multiuser.

Reviewed By:	bdrewery, imp
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D5683
2016-10-22 01:57:15 +00:00
Dimitry Andric
5763f79695 Merge ^/head r307383 through r307735. 2016-10-21 16:29:40 +00:00
Ed Maste
c615275ccc Don't build libssp as a prereq lib if WITHOUT_SSP is set
Reviewed by:	brooks
Differential Revision:	https://reviews.freebsd.org/D8301
2016-10-20 21:29:59 +00:00
Ed Maste
801b39194a Fix MK_SHAREDOCS test from r306864
Groff is needed only if MK_GROFF and MK_SHAREDOCS are both true.
2016-10-19 17:42:45 +00:00
Dimitry Andric
242b248284 Merge ^/head r306412 through r306905. 2016-10-09 13:30:57 +00:00
Baptiste Daroussin
24ff5a3223 groff is not needed in the bootstrap tools if the system is built
WITHOUT_SHAREDOCS

MFC after:	2 weeks
2016-10-08 18:57:11 +00:00
Ed Maste
335bcabec9 Merge ^/head r306303 through 306411. 2016-09-28 19:29:01 +00:00
Ed Maste
f987297fc9 Add a WITHOUT_DIALOG src.conf(5) knob
It also turns off dependencies (bsdinstall, bsdconfig, dpv, tzsetup).

Reviewed by:	dteske
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D7969
2016-09-27 18:08:38 +00:00
Dimitry Andric
2aeb03806a Merge ^/head r305220 through r305300. 2016-09-02 19:44:23 +00:00
Bryan Drewery
fcc803dfce Bump __FreeBSD_version for crunchgen META_MODE fix in r305254.
MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2016-09-01 23:22:31 +00:00
Dimitry Andric
986e05bc2a Completely revamp the way llvm, clang and lldb are built.
* Bootstrap llvm-tblgen and clang-tblgen with a minimal llvm static
  library, that has no other dependencies.
* Roll up all separate llvm libraries into one big static libllvm.
* Similar for all separate clang and lldb static libraries.
* For all these libraries, generate their .inc files only once.
* Link all llvm tools (including extra) against the big libllvm.
* Link clang and clang-format against the big libllvm and libclang.
* Link lldb against the big libllvm, libclang and liblldb.

N.B.: This is work in progress, some details may still be missing.

It also heavily depends on bsd.*.mk's support for SRCS and DPSRCS with
relative pathnames, which apparently does not always work as expected.
For building llvm, clang and lldb though, it seems to work just fine.

The main idea behind this restructuring is maintainability and build
peformance.  The previous large number of very small libraries, each
with their own generated files and dependencies was slow to traverse
and hard to understand.

Possible future improvements:
* Only build certain targets, e.g. for most regular users having just
  one target will be fine.  This will shave off some build time.
* Building the big llvm, clang and lldb libraries as shared (private)
  libraries.
* Adding other components from the LLVM project, such as lld.
2016-08-26 22:44:22 +00:00
Bryan Drewery
4d611b2b2d Fix in-tree GCC builds after r304681.
There were a few issues.
- In-tree GCC won't have X_COMPILER_TYPE defined but will have
  WANT_COMPILER_TYPE==gcc set from the SYSTEM_COMPILER logic that can
  be used.  Make the clang check specific to clang as well to ensure
  -target doesn't leak into a GCC build.
- When using a cross-compiler GCC (with a default sysroot or arch) and also
  passing --sysroot, it basically forgets all internal paths for
  libraries.  We've already worked around this quite a bit for
  the external toolchains.  Now for the in-tree bootstrap cross-compiler
  GCC, also pass in the needed -B${WORLDTMP}/usr/lib to find the crt
  object files, but also -isystem and -L to fix the paths.  This creates
  quite a spammy build log, but it is clear and still achieves the goals
  and stays consistent between internal and external build flags.
  Reducing the spam by using the '=' prefix feature will help and be
  done later.

MFC after:	3 days
X-MFC-With:	r304681
Reported by:	bz
Pointyhat to:	bdrewery
Sponsored by:	EMC / Isilon Storage Division
2016-08-23 19:29:37 +00:00
Bryan Drewery
bb0cd1b514 Always pass in -target and --sysroot flags for the build.
The internal bootstrap compiler has a default sysroot set by TOOLS_PREFIX
and target set by TARGET/TARGET_ARCH.  However, there are several needs to
always pass an explicit --sysroot and -target.
- External compiler needs sysroot and target flags.
- External ld needs sysroot.
- To be clear about the use of a sysroot when using the internal compiler.
- Easier debugging.
- Allowing WITH_SYSTEM_COMPILER+WITH_META_MODE to work together due to
  the flip-flopping build command when sometimes using external and
  sometimes using internal.
- Allow using no lld which has support for default paths.

The default sysroot in the bootstrap compiler is not changed.  The
buildenv compiler will still work with its default and will also
include -target/--sysroot from CC in the environment.

MFC after:	3 days
Discussed with:	emaste, brooks (BSDCam)
Reviewed by:	emaste
Sponsored by:	EMC / Isilon Storage Division
2016-08-23 15:20:32 +00:00
Bryan Drewery
0295d98c14 Trim unneeded bootstrap after r301470 made 9.1 the minimum supported release.
MFC after:	3 days
Sponsored by:	EMC / Isilon Storage Division
2016-08-16 12:13:12 +00:00
Bryan Drewery
c8de0c57d7 Avoid showing the bootstrap make command for check-old, etc.
Reported by:	koobs
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2016-08-12 13:52:51 +00:00
Li-Wen Hsu
14009f5e90 Only remove empty directories before packaging.
This preserves files are intentionally empty, most of them are in tests.txz

Reviewed by:	bdrewery
MFC after:	3 days
2016-08-10 18:22:42 +00:00
Mark Johnston
87ae82101f Add libprocstat prebuild library dependencies.
X-MFC-With:	r303531
2016-07-30 03:43:55 +00:00
Mark Johnston
a7e13d50fe librtld_db: Use the auxv to figure out where to look up loader symbols.
Previously, librtld_db just hardcoded /libexec/ld-elf.so, which isn't
correct for processes that aren't using the native ABI. With this change,
librtld_db can be used to inspect non-native processes; in particular,
dtrace -c now works for 32-bit executables on amd64.

MFC after:	1 month
2016-07-30 03:05:23 +00:00
Ed Maste
fd406aa3de Remove ${OBJDUMP} as it is not used by the base system
It was added to sys.mk relatively recently (r274503) for EFI builds
but is no longer used by the base system. The in-tree binutils are
outdated, will not be updated, and will be removed in the future.
Remove it from the toolchain build now to slightly simplify the build
and make sure we don't grow an accidental dependency.

Note that this affects only the toolchain build, and does not affect
/usr/bin/objdump in the built world.

Reviewed by:	bdrewery
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D6460
2016-07-27 14:58:15 +00:00
Bryan Drewery
7337126677 Don't disable binutils/elftoolchain bootstrapping with external compiler.
This was a regression from r300349.

Setting MK_CROSS_COMPILER=no forces the compiler bootstraping *and* the
binutils/elftoolchain bootstrapping to be disabled in share/mk/src.opts.mk.
The only intent with using an external compiler is to disable bootstrapping of
the compiler.  The binutils/elftoolchain bootstrapping must still occur unless
XAS is set.  This did not affect WITH_SYSTEM_COMPILER.

Now that setting an external compiler sets both MK_CLANG_BOOTSTRAP and
MK_GCC_BOOTSTRAP to no, and MK_CROSS_COMPILER does the same via
share/mk/src/opts.mk, remove redundant logic that checks for
MK_CROSS_COMPILER.  It will not always be true now that MK_CROSS_COMPILER==no
when an external compiler is used and --sysroot/-target is needed.

Reported by:	sbruno
Pointyhat to:	bdrewery
MFC after:	3 days
Sponsored by:	EMC / Isilon Storage Division
2016-07-24 18:50:11 +00:00
Bryan Drewery
a92545878a Fix empty WANT_COMPILER_TYPE when neither compiler is bootstrapped.
Bug in r303272.

MFC after:	3 days
X-MFC-With:	r303272
Sponsored by:	EMC / Isilon Storage Division
2016-07-24 18:28:14 +00:00
Bryan Drewery
b17f5b2261 SYSTEM_COMPILER: Rework the logic to allow a 'make test-system-compiler'.
1. Always calculate what the expected values are.
2. Add 'make test-system-compiler' to show all of the computed values
   vs the wanted values.
3. Extend the .info line to buildkernel/kernel-toolchain/toolchain/_cross-tools.
4. Consolidate all of the logic to one condition.

MFC after:	3 days
Sponsored by:	EMC / Isilon Storage Division
2016-07-24 18:05:05 +00:00
Bryan Drewery
4143850b00 Add LOCAL_XTOOL_DIRS to add into cross-tools.
Sponsored by:	EMC / Isilon Storage Division
2016-07-21 06:06:11 +00:00
Ed Maste
fe50ba0583 Include makewhatis in ITOOLS when MK_MAN_UTILS is true
Previously it was conditional on MK_MAN. It's possible to build
FreeBSD with man pages but without man page tools. MK_MAN_UTILS
is the conditional used in share/man/Makefile for determining whether
makewhatis is executed at install time, so it is the proper one for
ITOOLS as well.

PR:		210142
MFC after:	1 week
2016-07-19 18:05:25 +00:00
Cy Schubert
bcc238eaed Revert r302865 and it's sister r302866 as r302865 causes some breakage.
Reported by:	Guy Yur <guyyur@gmail.com>
2016-07-15 15:09:48 +00:00
Cy Schubert
507cef9302 When building multiple kernels using KERNCONF, non-existent KERNCONF
files will produce an error and buildkernel will fail. Previously missing
KERNCONF files silently failed giving no indication as to why, only to
subsequently discover during installkernel that the desired kernel was
never built in the first place.

Reviewed by:		ngie@
MFC after:		1 week
Differential Revision:	D7167
2016-07-15 04:43:38 +00:00
Bryan Drewery
b987685fe0 Revert r302670 and r302671 for now.
MACHINE_CPUARCH smells like MACHINE except for arm64/aarch64 which
has it backwards.
2016-07-13 01:35:53 +00:00
Bryan Drewery
79156eb7c9 Create a TARGET_CPUARCH thing to go with MACHINE_CPUARCH.
MFC after:	3 days
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D7160
2016-07-12 18:57:28 +00:00
Glen Barber
6cdb0cba3e Fix TARGET_TRIPLE for 12.0-CURRENT.
Submitted by:	rene
Sponsored by:	The FreeBSD Foundation
2016-07-11 21:55:56 +00:00
Bryan Drewery
a356b78462 WITH_SYSTEM_COMPILER: Don't enable for xdev or native-xtools.
Otherwise they get no compiler.

Reported by:	swills
Tested by:	swills
Approved by:	re (gjb)
Sponsored by:	EMC / Isilon Storage Division
2016-07-07 21:03:50 +00:00
Bryan Drewery
48413367ef WITH_SYSTEM_COMPILER: Avoid using GCC4.8+ logic on native GCC4.2 platforms.
As the XXX notes, these should really be checking MK_GNUCXX since there is
already a version check in share/mk/src.opts.mk to disable it.  Fixing that
here is more complex though.  This could also be using X_COMPILER_FEATURES
but uses X_COMPILER_VERSION to keep in sync with the src.opts.mk logic.

Tested by:	andreast
Sponsored by:	EMC / Isilon Storage Division
Approved by:	re (gjb)
2016-06-29 01:50:30 +00:00
Bryan Drewery
0739409184 Fix native powerpc64 build of lib32 with in-tree GCC.
- This was broken by r300350 and r300885.
- Add some comments around the external GCC logic since it is spread out
  and in need of some cleanup.
- The problem was that X_COMPILER_TYPE is always defined from CC->XCC's
  default, so if /usr/bin/cc is GCC (as it is on native powerpc64) then
  X_COMPILER_TYPE was getting GCC and triggering the external logic in
  Makefile.libcompat.  It was intended to always provide -isystem with
  GCC since --sysroot is used into the lib32 sysroot which won't modify
  the header path without the -isystem.  The use of the libc++/std=c++11
  override was only intended to be used for external compilers though
  (more accurately GCC 4.8+ but that's a separate assumption to
  cleanup).  Apply the same logic from Makefile.inc1 to Makefile.libcompat
  to only add the libc++ override when needed for external compilers.

Pointyhat to:	bdrewery
Tested with:	native ppc64 (swills), universe, ppc64 xtoolchain,
		amd64 xtoolchain, sparc64 cross-build of ppc64 (host GCC 4.2)
Reported by:	andreast, swills
Approved by:	re (gjb)
Sponsored by:	EMC / Isilon Storage Division
2016-06-15 23:58:03 +00:00
Jilles Tjoelker
d80f1dd1d7 build: Add legacy support for futimens() and utimensat().
In order to allow using utimensat() in install(1), add futimens() and
utimensat() to -legacy.

The files futimens.c and utimensat.c are modified copies of the files under
lib/libc/sys/ since the libc versions use symbols that do not exist in the
libc on the build system (sys_futimens and sys_utimensat) . I expect the
next non-sweeping change to both sets of files to be to delete them, anyway.

This will allow reverting r299942 (which is a revert of r299850) enabling
nanosecond timestamps in install(1).

Reviewed by:	bdrewery
2016-06-09 21:57:34 +00:00
Bryan Drewery
aa06dfc9c3 legacy: Avoid building/installing headers twice.
Sponsored by:	EMC / Isilon Storage Division
2016-06-05 23:56:28 +00:00
Bryan Drewery
4d40df1bff Add a MINIMUM_SUPPORTED_OSREL and bump it to 900044.
This is actually a revision in the stable/9 branch released as 9.1.  The
localedef build requires xlocale from this period.

Sponsored by:	EMC / Isilon Storage Division
2016-06-05 23:05:10 +00:00
Bryan Drewery
95348a57d0 Fix incorrect logic in r301394.
Reported by:	Mark Millard
2016-06-04 17:38:17 +00:00
Bryan Drewery
b4dd422926 Follow-up r301287: Pass external compiler metadata when used.
This fixes WITH_SYSTEM_COMPILER, WITHOUT_CROSS_COMPILER, external compiler, to
pass the external compiler metadata rather than the ${CC} metadata.  On
a build host that has clang as CC it was passing the clang metadata rather than
GCC metadata during the build.
2016-06-04 17:17:10 +00:00
Bryan Drewery
bbfcc681bc WITHOUT_CROSS_COMPILER: Fix installworld.
Since no WORLDTMP/usr/bin/cc is created, cc cannot be found
during installworld time since /usr/bin is not in the PATH.
Pass along the known compiler metadata to allow installworld
to work.  The same fix was used for WITH_SYSTEM_COMPILER.

A better route would be to store a cookie in buildworld
containing this compiler metadata and then using that
at install time, rather than rerunning cc.

Reported by:	Mark Millard
Sponsored by:	EMC / Isilon Storage Division
2016-06-03 20:49:49 +00:00
Dimitry Andric
c371433e4a For clang, move the definition of FREEBSD_CC_VERSION into its own header
file, lib/clang/freebsd_cc_version.h, instead of reusing Version.inc.
The header is only included from one .cpp file in the clang tree.

This minimizes the number of .cpp files that need to be rebuilt if the
version is bumped.

Discussed with:	bdrewery
2016-06-03 16:17:36 +00:00
Bryan Drewery
f661dbee81 GCC External: Revert r300886, r300904, r300917, r300918
The fix in r300873 is mostly enough.  A fix for lib32 will be
committed.separately.
2016-05-29 06:20:15 +00:00
Bryan Drewery
ce00342bb2 Move external GCC compiler hacks to bsd.sys.mk.
This allows respecting -nostdinc, -nostdinc++ and -nostdlib before
making the decision to add in -isystem, etc.  The -isystem flags
are problematic for building lib/libc++ and lib/libcxxrt which wants
to only use its own headers.

More information the need of these flags can be found at
https://gcc.gnu.org/ml/gcc/2016-03/msg00219.html

This also reverts r300873.

Sponsored by:	EMC / Isilon Storage Division
2016-05-27 23:03:44 +00:00
Glen Barber
5929ee2961 Make PKG_VERSION aware of '-ALPHA*'.
Submitted by:	Ben Woods
Sponsored by:	The FreeBSD Foundation
2016-05-27 20:33:14 +00:00
Bryan Drewery
eb9b07bb77 WITH_AUTO_OBJ: Fix crunchgen builds.
Passing MAKEOBJDIRPREFIX to the main prog build (rescue) would confuse
WITH_AUTO_OBJ and cause it to create a recursed object directory that
then broke the actual prog build.  This is normally not a problem since
we do not call 'make -f prog.mk obj' before building anything in it.

Crunchgen(1) also assumes that if -o is not passed then if an object
directory does not already exist then it should build in the source
directories.  The normal buildworld process will have already ran 'make
obj' in each of the component directories so this is not a problem.
With WITH_AUTO_OBJ though this is not the case.  So we must tell
crunchgen(1) that MK_AUTO_OBJ=yes will create the directory and to not
require it be present before generating its Makefile.

Sponsored by:	EMC / Isilon Storage Division
2016-05-26 23:20:36 +00:00
Bryan Drewery
016448a507 WITH_META_MODE: Move the kernel support to kern.pre.mk.
This allows using META_MODE directly from the kernel build directory.
This also allows removing a hack from the DIRDEPS_BUILD kernel target.

Sponsored by:	EMC / Isilon Storage Division
2016-05-26 23:20:01 +00:00
Bryan Drewery
2d86a958e2 Remove leftover _crunchide from r283108 2016-05-26 23:19:57 +00:00
Dimitry Andric
9729cf0934 Update libc++ to 3.8.0. Excerpted list of fixes (with upstream revision
numbers):

r242679 Implement the plugin-based version of std::search. There are no
        searchers yet; those are coming soon.
r242682 Implement the default searcher for std::experimental::search.
r243728 Add <experimental/any> v2.
r245330 implement more of N4258 - Cleaning up noexcept in the standard
        library. Specifically add new noexcept stuff to vector and
	string's move-assignment operations
r245334 Fix PR22606 - Leak pthread_key with static storage duration to
        ensure all of thread-local destructors are called.
r245335 Fix PR23589: std::function doesn't recognize null pointer to
        varargs function.
r247036 Implementation of Boyer-Moore and Boyer-Moore-Horspool
        searchers for the LFTS.
r249325 Implement LWG#2063, and update the issues links to point to the
        github generated pages
r249738 Split <ctype.h> out of <cctype>.
r249739 Split <errno.h> out of <cerrno>.
r249740 Split <float.h> out of <cfloat>.
r249741 Split <inttypes.h> out of <cinttypes>.
r249742 Split <math.h> out of <cmath>.
r249743 Split <setjmp.h> out of <csetjmp>.
r249761 Split <stddef.h> out of <cstddef>.
r249798 Split <stdio.h> out of <cstdio>.
r249800 Split <stdlib.h> out of <cstdlib>.
r249889 Split <wchar.h> out of <cwchar>.
r249890 Split <wctype.h> out of <cwctype>.
r249929 Split <string.h> out of <cstring>.
r250254 ABI versioning macros for libc++.
r251246 Fix LWG#2244: basic_istream::seekg
r251247 Fix LWG#2127: Move-construction with raw_storage_iterator.
r251253 Fix LWG#2476: scoped_allocator_adaptor is not assignable
r251257 Fix LWG#2489: mem_fn() should be noexcept
r251618 Implement P0004R1 'Remove Deprecated iostreams aliases'
r251766 Implement the first part of P0006R0: Adopt Type Traits Variable
        Templates for C++17.
r252195 Implement P0092R1 for C++1z
r252350 Allow deque to handle incomplete types.
r252406 More of P0006R0: type traits variable aliases for C++17.
r252407 Implement LWG#2353: std::next is over-constrained
r252905 Implement P0074: Making owner_less more flexible
r253215 Implement P0013R1: Logical Operator Type Traits.
r253274 Implement P0007: Constant View: A proposal for a std::as_const
        helper function template.
r254119 Add static_assert to set/multiset/map/multimap/forward_list/deque
        that the allocator's value_type match the container's value_type.
r254283 Implement more of P0006; Type Traits Variable Templates.
r255941 LWG2485: get() should be overloaded for const tuple&&.
r256325 Fix LWG Issue #2367 - Fixing std::tuple and std::pair's default
        constructors.
r256652 Fix for ALL undefined behavior in <list>.
r256859 First half of LWG#2354: 'Unnecessary copying when inserting
        into maps with braced-init syntax'

Exp-run:	antoine
Relnotes:	yes
2016-05-26 18:52:49 +00:00
Bryan Drewery
7e1b3c5e2c Move binutils handling after src.opts.mk.
This fixes the arm64 build after r300348.

Sponsored by:	EMC / Isilon Storage Division
2016-05-23 16:24:34 +00:00
Bryan Drewery
6120aabd32 Opportunistically skip building a cross-compiler with SYSTEM_COMPILER set.
This will still build the compiler for the target but will not build the
bootstrap cross-compiler in the cross-tools phase.  Other toolchain
bootstrapping, such as elftoolchan and binutils, currently still occurs.

This will utilize the default CC (cc, /usr/bin/cc) as an external compiler.

This is planned to be on-by-default eventually.

This will utilize the __FreeBSD_cc_version compiler macro defined in the
source tree and compare it to CC's version.  If they match then the
cross-compiler is skipped.  If [X]CC is an external compiler (absolute
path) or WITHOUT_CROSS_COMPILER is already set, then this logic is skipped.
If the expected bootstrap compiler type no longer matches the found CC
compiler type (clang vs gcc), then the logic is skipped.  As an extra
safety check the version number is also compared from the compiler to
the tree version.

Clang:
  The macro FREEBSD_CC_VERSION is defined in:
    lib/clang/include/clang/Basic/Version.inc
  For clang -target will be used if TARGET_ARCH != MACHINE_ARCH.  This
  is from the current external toolchain logic.  There is currently an
  assumption that the host compiler can build the TARGET_ARCH.  This
  will usually be the case since we don't conditionalize target arch
  support in clang, but it will break when introducing new
  architectures.  This problem is mitigated by incrementing the version
  when adding new architectures.

GCC:
  The macro FBSD_CC_VER is defined in:
    gnu/usr.bin/cc/cc_tools/freebsd-native.h
  For GCC there is no simple -target support when TARGET_ARCH !=
  MACHINE_ARCH.  In this case the opportunistic skip is not done.  If we
  add proper support for this case in external toolchain logic then it
  will be fine to enable.

This relies on the macros being incremented whenever any change occurs
to these compilers that warrant rebuilding files.  It also should never
repeat earlier values.

Reviewed by:	brooks, bapt, imp
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D6357
2016-05-21 01:32:23 +00:00
Bryan Drewery
76286f4bc5 Enable external compiler logic if both bootstrap compilers are disabled.
Reviewed by:	brooks
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D6358
2016-05-21 01:32:16 +00:00
Bryan Drewery
c1b9d0022d Enable and utilize WITHOUT_CROSS_COMPILER logic for external CC.
This is a NOP.

Reviewed by:	brooks, bapt
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D6354
2016-05-21 01:32:07 +00:00
Bryan Drewery
34fb4407a8 Move external toolchain support earlier.
This is to consolidate external toolchain and WITHOUT_CROSS_COMPILER support.

Reviewed by:	brooks, bapt
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D6353
2016-05-21 01:32:04 +00:00
Bryan Drewery
bf0c0f96cf WITH_META_MODE: Use META_MODE rather than .MAKE.MODE for buildkernel.
This is mostly a style change so that other code does not duplicate
it.  The problem is when META_MODE needs to be disabled but it has
been overridden by .MAKE.MODE.

Sponsored by:	EMC / Isilon Storage Division
2016-05-21 01:31:51 +00:00
Warner Losh
e873d4db28 Turns out, mipsn32el worked at one point, so add it back in.
Pointed out by: jmallet@
2016-05-19 02:58:30 +00:00
Warner Losh
14e5b32daf Fix negative logic error. We want gnueabihf ABI when we're NOT
requesting soft float ABI.

Noticed by: bdrewery
2016-05-18 22:50:25 +00:00
Warner Losh
631dbc3b78 mipsn32el is not a thing, remove it from the list of valid
architectures. There's no definition for it, nobody uses it
and it is unlikely to ever work. We can put it back when someone
demonstrates it running...

The new default for armv6 is hard float, so extend that default
to the external toolchain support.
2016-05-18 20:49:09 +00:00
Bryan Drewery
ecb92e2d47 Combine restage/reinstall in a safe way.
Sponsored by:	EMC / Isilon Storage Division
2016-05-18 17:08:29 +00:00
Warner Losh
2c0e9e2a09 Make armv6 hard float abi by default. Kill armv6hf.
Allow CPUTYPE=soft to build the current soft-float abi libraries.
Add UPDATING entry to announce this.

Approved by: re@ (gjb)
2016-05-18 06:01:18 +00:00
Bryan Drewery
396f5ab6ee Support libsoft for restage.
This is essentially fixing a merge conflict.
2016-05-13 17:44:20 +00:00
Glen Barber
d4ca548306 Revert r299292, which was not a mismerge. It has an additional horrible
side effect of replacing /etc/{master.,}passwd and /etc/group.

Submitted by:	O. Hartmann
Pointyhat to:	gjb (myself)
Sponsored by:	The FreeBSD Foundation
2016-05-10 05:47:25 +00:00
Glen Barber
381fcf101d Combine the 'reinstall' and 'restage' targets, which appears to have
been a mismerge.

Sponsored by:	The FreeBSD Foundation
2016-05-10 00:47:49 +00:00
Enji Cooper
79ccc589d2 Default NO_INSTALLEXTRAKERNELS to yes, not no
The old (^/stable/9) default was yes, not no ("no" was the new default
introduced recently that broke POLA). Restore it to keep POLA like
glebius intended in r299077

MFC after: 3 days
X-MFC with: r299086
Pointyhat to: ngie (research before assuming and committing next time)
Sponsored by: EMC / Isilon Storage Division
2016-05-04 21:15:28 +00:00
Enji Cooper
c0fd425526 Default NO_INSTALLEXTRAKERNELS to "no" to unbreak the build
MFC after: soon (was insta-MFCed -_-..)
Pointyhat to: glebius
Sponsored by: EMC / Isilon Storage Division
2016-05-04 21:00:41 +00:00
Gleb Smirnoff
f4ae78bd4e Make it possible to override NO_INSTALLEXTRAKERNELS.
Reviewed by:	gjb
2016-05-04 17:21:34 +00:00
Pedro F. Giffuni
a1ff7af013 Misc. build: minor spelling fixes.
No functional change.
2016-05-03 22:01:48 +00:00
Bryan Drewery
561e173f7e WITH_META_MODE: Allow buildkernel to create .meta files with curdir==objdir.
Without this the incremental build was broken since .depend.* are not
generated with .MAKE.MODE=meta and .meta files were not created to
track dependencies.  Typically meta mode does not create .meta files
when building with curdir==objdir but the kernel build is special.

Reported by:	Nikolai Lifanov <lifanov@mail.lifanov.com>
Sponsored by:	EMC / Isilon Storage Division
2016-04-26 18:08:51 +00:00
Bryan Drewery
75a6cb4f34 Fix unset variables from r298417.
Pointyhat to:	bdrewery
Sponsored by:	EMC / Isilon Storage Division
2016-04-21 16:50:45 +00:00
Bryan Drewery
40cf3344dc Add more missing .PHONY
Sponsored by:	EMC / Isilon Storage Division
2016-04-21 16:43:36 +00:00
Bryan Drewery
59e2f057b6 Remove redundant logic from the pkg-base merge.
These will be set from the logic right above it.

Reviewed by:	gjb
Sponsored by:	EMC / Isilon Storage Division
2016-04-21 16:30:28 +00:00
Bryan Drewery
ae00149999 Fix 'make -n' for new packaging targets.
Sponsored by:	EMC / Isilon Storage Division
2016-04-21 16:30:25 +00:00
Bryan Drewery
c943a53eb4 Add more missing .PHONY.
This also protects them from trying to create .meta files
with WITH_META_MODE.

Reported by:	Nikolai Lifanov <lifanov@mail.lifanov.com>
Sponsored by:	EMC / Isilon Storage Division
2016-04-21 16:30:16 +00:00
Ed Maste
97c316c68c Adjust BOOTSTRAPPING test after MFC of kbdcontrol's -P option 2016-04-21 13:14:55 +00:00
Glen Barber
fcede88bbd Add MK_AUTO_OBJ=no to ${MAKE} invocation if PKG_VERSION needs to be
created to avoid creating ${.OBJDIR}.

The duplicate REVISION/BRANCH/VERSION evaluation will be addressed
separately.

Sponsored by:	The FreeBSD Foundation
2016-04-18 19:27:51 +00:00
Glen Barber
0edd2576c0 MFH
Sponsored by:	The FreeBSD Foundation
2016-04-16 02:32:12 +00:00
Bryan Drewery
473fda75dd META_MODE+filemon: Default -DNO_CLEAN enabled.
When using meta mode with filemon, the build is reliably incremental
safe.  Bmake will use the meta files, along with filemon information,
to rebuild targets when their dependencies change, commands change,
or files they generate are missing.

Sponsored by:	EMC / Isilon Storage Division
2016-04-14 22:00:33 +00:00
Bryan Drewery
fdff4951ca META_MODE: Pass along the default sysroot in bootstrap-tools to avoid rebuilds later.
Some of the clang libraries build in this phase and the cross-tools
phase.  Later in the cross-tools phase when they build with a default
TOOLS_PREFIX, they see a changed build command in meta mode due to
the changed DEFAULT_SYSROOT.  This is avoided by passing along
TOOLS_PREFIX earlier.

Sponsored by:	EMC / Isilon Storage Division
2016-04-14 21:09:03 +00:00
Bryan Drewery
d1dd034d07 META_MODE: Don't rebuild build-tools targets during normal build.
This avoids 'build command changed' due to CFLAGS/CC changes during the
normal build.  Without this the build-tools targets end up rebuilding
for the *target* rather than keeping the native versions built in
build-tools.

Sponsored by:	EMC / Isilon Storage Division
2016-04-14 21:06:10 +00:00
Bryan Drewery
a561b1060c Mark some more .PHONY targets.
Sponsored by:	EMC / Isilon Storage Division
2016-04-14 21:04:37 +00:00
Bryan Drewery
7cbd0a2953 Simplify building libpam and fix libpam.a not containing the modules since r284345.
The change in r284345 moved the creation of openpam_static_modules.o to
lib/libpam/static_modules but never managed to get them into libpam.a.

Move this logic to lib/libpam/static_libpam and have it create a static
library for libpam.a  The main lib/libpam/libpam will only create a
shared library.  No redundancy in compilation or installation exists
in this solution.

This avoids requiring a pass with -D_NO_LIBPAM_SO_YET.

Sponsored by:	EMC / Isilon Storage Division
2016-04-14 01:17:03 +00:00
Glen Barber
9c831bbd69 MFH
Sponsored by:	The FreeBSD Foundation
2016-04-13 02:04:09 +00:00
Steve Wills
d13c461400 Try harder to find svn
While here, elliminate last references to CVS_UPDATE and SUP_UPDATE

Reviewed by:	gjb
Approved by:	gjb
2016-04-13 01:47:04 +00:00
Glen Barber
14cda706cb Set explicit ordering to prevent packaging failures with parallel
(-jN) make(1) jobs.

Sponsored by:	The FreeBSD Foundation
2016-04-11 18:10:20 +00:00
Glen Barber
7a888f325e Avoid using DESTDIR in packaged-base targets, which can
cause bizarre packaging failures with non-default OBJDIR
and/or MAKEOBJDIRPREFIX and REPODIR set to a location not
within OBJDIR.

Reported by:	many
Tested by:	sef
Sponsored by:	The FreeBSD Foundation
2016-04-08 01:57:40 +00:00
Glen Barber
2263fb580e MFH
Sponsored by:	The FreeBSD Foundation
2016-04-06 01:44:21 +00:00
Glen Barber
31f3abb19c Fix errors packaging the base system when using non-default DESTDIR
and MAKEOBJDIRPREFIX.

Sponsored by:	The FreeBSD Foundation
2016-04-05 23:24:17 +00:00
Bryan Drewery
91da76981b Remove leftover _LDSCRIPTROOT missed in r297270.
Sponsored by:	EMC / Isilon Storage Division
2016-04-05 21:12:03 +00:00
Glen Barber
d60840138f MFH
Sponsored by:	The FreeBSD Foundation
2016-04-04 23:55:32 +00:00
Bryan Drewery
d043a786e8 Fix the external GCC build after r297271 by setting -L <sysroot>/usr/lib.
GCC does add <sysroot>/usr/lib to the library search path but it comes after
/usr/local/lib which can find ports libraries such as libedit.so.  The
bad path comes in as /usr/local/lib/gcc/x86_64-portbld-freebsd11.0/5.3.0/../../../
which corresponds to <prefix>/lib.

This partially reverts r297271.

Pointyhat to:	bdrewery
Sponsored by:	EMC / Isilon Storage Division
2016-03-30 23:50:29 +00:00
Bryan Drewery
497e80911e Remove the old depend (mkdep) code and make FAST_DEPEND the one true way.
Reviewed by:	emaste, hselasky (partial), brooks (brief)
Discussed on:	arch@
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D5742
2016-03-30 23:50:23 +00:00
Ed Maste
915d57ae35 simplify compile-time default keyboard map generation
In r296926 the -P <path> option was added to kbdcontrol, which enables
this change for a simplified compile-time default keymap build process.

PR:		193865
Reviewed by:	Oliver Pinter
Tested by:	Oliver Pinter
MFC After:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D5708
2016-03-28 19:51:45 +00:00
Bryan Drewery
ac978f5a8f WITHOUT_TOOLCHAIN: Fix includes not being staged in WORLDTMP.
This has been the case since r264930 and r274662.

Sponsored by:	EMC / Isilon Storage Division
2016-03-25 22:36:32 +00:00
Bryan Drewery
4e3252c407 CROSS_BINUTILS_PREFIX: Reduce redundant logic.
Sponsored by:	EMC / Isilon Storage Division
2016-03-25 22:36:26 +00:00
Bryan Drewery
5df2c3ccc4 External compiler: Remove redundant flags from CXXFLAGS.
The use of XCXXFLAGS is to assign it to CXX in CROSSENV.
XCFLAGS is also assigned here so there is no need to have
--syroot and -B flags again.

Sponsored by:	EMC / Isilon Storage Division
2016-03-25 22:36:23 +00:00
Bryan Drewery
d63cb012ba WITHOUT_CROSS_COMPILER: Fix this to use external compiler logic.
Without this the default toolchain in /usr/bin/ would not use
WORLDTMP via --sysroot, and would lack --target if cross-building.

PR:		196193
Related:	D3970
Sponsored by:	EMC / Isilon Storage Division
2016-03-25 22:36:13 +00:00
Bryan Drewery
1b89301d3b Update flags for external GCC.
- The -L WORLDTMP/usr/lib is not needed as GCC is already adding in
  -L =/usr/lib internally with --sysroot.  It does not do this for
  header include paths though, thus passing -isystem =/usr/include is
  still needed.

For the forced libc++ usage:
- Use -isystem rather than -I for libc++ headers.
- Use -std=c++11 rather than gnu++11.
- Use -nostdinc++ to ensure GCC's headers don't leak in.

Sponsored by:	EMC / Isilon Storage Division
2016-03-25 19:12:41 +00:00
Bryan Drewery
1b337a349d Build libcompat (lib32) with a --sysroot pointing into its stage directory.
This overrides the cross-compiler's default sysroot to use the WORLD32's
sysroot for building the lib32 libraries.  Previously the cross-compiler
would default the sysroot to the 64bit WORLDTMP and -B/-L/-isystem flags
were used to build using the lib32 files.  This leads to multiple issues
discussed later.  Some extra headers are now needed to be staged since the
64bit WORLDTMP is not referenced at all for headers.  The 64bit WORLDTMP
is still used via PATH for build tools.  Overriding the default
target/arch is retained in the CC/CXX overrides.

This allows reverting the LDSCRIPT rewriting in installworld from r296921 and
r235122, thus allowing read-only objdirs to work for installing again.

This removes the need for _LDSCRIPTROOT.

This allows progressing the change to always use --sysroot for the build
rather than only relying on the cross-compiler's default sysroot.  The
work for that is in D3970 and needed to resolve WITHOUT_CROSS_COMPILER
not using a --sysroot [1].

PR:		196193 [1]
Sponsored by:	EMC / Isilon Storage Division
2016-03-25 19:12:38 +00:00
Glen Barber
b0b644948c Make WSTAGEDIR, KSTAGEDIR, and REPODIR configurable.
This should fix the 'packages' target when MAKEOBJDIRPREFIX or
DESTDIR is set in the make(1) environment or via command line.

Reported by:	kmoore, sef, Marko Turk
Sponsored by:	The FreeBSD Foundation
2016-03-22 21:02:04 +00:00
Glen Barber
538354481e MFH
Sponsored by:	The FreeBSD Foundation
2016-03-14 18:54:29 +00:00
Warner Losh
bb52d711d0 Use the newly minted Makefile.libcompat to implement libsoft libraries
for the armv6 ABI switch. This also make WITH_LIBSOFT functional on
the arm platform. As a transition thing, this seems to work even
without switching the ABI (we basically build the same libraries
twice when MK_LIBSOFT=yes until the ABI cut over next
month). MK_LIBSOFT remains default no.
2016-03-12 23:25:05 +00:00
Ed Maste
cb1469fdd6 Bump BOOTSTRAPPING test for libelf after cross-endian fix in r296685 2016-03-12 19:08:13 +00:00
Bryan Drewery
dba75e9fa5 Move Makefile.lib32 to Makefile.libcompat and generalize it.
This is in preparation for LIBSOFT.

This file only supports *1* LIBCOMPAT value currently and must be capitalized.
In Makefile.libcompat given LIBCOMPAT=FOO there can be values set for
LIBFOOCFLAGS, LIBFOOCPUFLAGS, LIBFOOWMAKEENV, LIBFOOWMAKEFLAGS, LIBFOOCPUFLAGS,
and LIBFOODTRACE.  These will have the standard cross-build values appended
onto them.

This could be extended to support multiple libcompat libraries in the future
once there is a need.

Reviewed by:	imp
Sponsored by:	EMC / Isilon Storage Division
Differential Revision:	https://reviews.freebsd.org/D5612
2016-03-11 23:56:33 +00:00
Warner Losh
5202371725 Factor out lib32 generation to its own file. This is prep for a
similar Makefile.libsoft which will do the same for armv6 soft fp API
libraries in prep for pulling the trigger on moving to armv6 hard
float. Once there's two files, I'll work with bdrewery@ to merge the
two files as they are mostly the same. The high rate of churn for
Makefile* makes it quite difficult to make progress out of tree.

Differential Review: https://reviews.freebsd.org/D5566
2016-03-11 03:38:10 +00:00
Glen Barber
7d536dc855 MFH
Sponsored by:	The FreeBSD Foundation
2016-03-10 21:16:01 +00:00
Bryan Drewery
0481f7852d Don't ever create object directories here with MK_AUTO_OBJ.
Sponsored by:	EMC / Isilon Storage Division
2016-03-08 21:26:25 +00:00
Glen Barber
b0bdd78348 Avoid adding the '-release' suffix to non-debug kernel packages.
Sponsored by:	The FreeBSD Foundation
2016-03-08 19:35:30 +00:00
Glen Barber
b655ec9752 MFH
Sponsored by:	The FreeBSD Foundation
2016-03-06 04:13:17 +00:00
Dimitry Andric
a36b180a7f Merge ^/head r296369 through r296409. 2016-03-05 15:34:54 +00:00
Bryan Drewery
34f4acc8a8 Conditionalize building makewhatis(1) for upgrades that need it.
Remove building of the legacy makewhatis(1) since it was only needed for
6.0 upgrades.

On my 2.2 GHz system libsqlite3 takes 60-100 seconds to build, which due to
its serialized nature can hold up the build waiting on it to finish in
bootstrap-tools.

makewhatis(1) was only required to be a build tool to support upgrades
from 6.0 which was properly removed in r208324 but later reverted due to
installworld using it.  The installworld issue was fixed in r275622
when it was added to ITOOLS.  A BOOTSTRAPPING check was missed when
makewhatis(1) was replaced with mandoc in r283777.

Sponsored by:	EMC / Isilon Storage Division
2016-03-05 05:08:57 +00:00
Bryan Drewery
2bece0ce3d Don't add duplicates from LOCAL_LIB_DIRS.
This causes parallel failures.

Sponsored by:	EMC / Isilon Storage Division
2016-03-04 22:37:16 +00:00
Bryan Drewery
400b94bbeb Fix over/under-linking in contrib/ofed.
Sponsored by:	EMC / Isilon Storage Division
2016-03-04 22:37:09 +00:00
Bryan Drewery
bd6daf52d3 NO_ROOT: Create the METALOG directory on installworld/distributeworld.
When not using NO_ROOT the DESTDIR is implicitly created by the
distrib-dirs call.  However with NO_ROOT it is trying to write
to the METALOG right away before running distrib-dirs which
may fail.

Sponsored by:	EMC / Isilon Storage Division
2016-03-04 22:36:56 +00:00
Ed Maste
6cb64f8497 Add dd as a bootstrap tool, for status= support
In r295757 I added status=none to buildworld's use of dd, but FreeBSD
versions before April 2014 lack support.
2016-03-04 16:03:15 +00:00
Dimitry Andric
82aa34e6fa Merge ^/head r296007 through r296368. 2016-03-03 23:15:46 +00:00
Glen Barber
28c9d1af05 Set ALLOW_BASE_SHLIBS when creating base packages to enable
shared library tracking.

Note, this requires a patched pkg(8) to do anything, but pkg(8)
does not complain when an option is specified that it does not
recognize.

Sponsored by:	The FreeBSD Foundation
2016-03-02 21:04:17 +00:00
Glen Barber
52259a98ad MFH
Sponsored by:	The FreeBSD Foundation
2016-03-02 16:14:46 +00:00
Mariusz Zaborski
c501d73c7e Convert casperd(8) daemon to the libcasper.
After calling the cap_init(3) function Casper will fork from it's original
process, using pdfork(2). Forking from a process has a lot of advantages:
1. We have the same cwd as the original process.
2. The same uid, gid and groups.
3. The same MAC labels.
4. The same descriptor table.
5. The same routing table.
6. The same umask.
7. The same cpuset(1).
From now services are also in form of libraries.
We also removed libcapsicum at all and converts existing program using Casper
to new architecture.

Discussed with:		pjd, jonathan, ed, drysdale@google.com, emaste
Partially reviewed by:	drysdale@google.com, bdrewery
Approved by:		pjd (mentor)
Differential Revision:	https://reviews.freebsd.org/D4277
2016-02-25 18:23:40 +00:00
Dimitry Andric
14e9c9161c Merge ^/head r295902 through r296006. 2016-02-24 21:38:51 +00:00
Bryan Drewery
f4d0f67fc9 FAST_DEPEND: Skip 'make depend' for buildworld and kernel since it is auto-ran now.
Sponsored by:	EMC / Isilon Storage Division
2016-02-24 17:20:22 +00:00
Glen Barber
3a3f435f98 Include a 'package-pkg' target, intended for use for
architectures we do not provide upstream pkg(8) packages.

This is not tied to anything as-is, and likely will break
your system if used (based on experience with testing with
powerpc).

There is an overwhelming amount of evil happening here,
so until the issues are fixed, it will not be tied into the
'packages' target.

Sponsored by:	The FreeBSD Foundation
2016-02-24 03:05:05 +00:00
Glen Barber
9096340371 Set PORTSDIR if not already set.
Sponsored by:	The FreeBSD Foundation
2016-02-24 03:00:06 +00:00
Glen Barber
debac084e6 Further subdir-ize WSTAGEDIR and KSTAGEDIR base on the
TARGET/TARGET_ARCH value, since there are valid use cases
for having multiple on any given system.

Sponsored by:	The FreeBSD Foundation
2016-02-24 02:59:14 +00:00
Dimitry Andric
9893f787ec Merge ^/head r295601 through r295844. 2016-02-21 13:49:26 +00:00
Glen Barber
72c3aa02dc MFH
Sponsored by:	The FreeBSD Foundation
2016-02-18 00:37:58 +00:00
Glen Barber
4821f9ed5b Always include the kernel name in the resulting package.
While functionally expected to be a no-op on big-iron hardware,
embedded hardware (arm, mips) do not have a GENERIC kernel, so
the KERNCONF value must be included in the package to avoid
conflicting packages for the default kernel (RPI-B versus RPI2,
for example).

While here, correct the kernel name in the metadata.

Sponsored by:	The FreeBSD Foundation
2016-02-16 20:11:00 +00:00
Bryan Drewery
619a5d7bac Don't descend into test directories for distribution->installconfig.
Sponsored by:	EMC / Isilon Storage Division
2016-02-16 02:14:09 +00:00
Glen Barber
0fe0fe112f MFH
Sponsored by:	The FreeBSD Foundation
2016-02-15 21:58:52 +00:00
Dimitry Andric
90a4726375 Merge ^/head r295544 through r295600. 2016-02-13 16:02:12 +00:00
Ed Maste
c1da297433 Add libpe for elfcopy(1) PE/COFF support
Sponsored by:	The FreeBSD Foundation
2016-02-12 21:12:47 +00:00
Dimitry Andric
4156ce4fed Merge ^/head r295351 through r295543. 2016-02-11 20:07:09 +00:00
Glen Barber
acf5bd83be Remove unnecessary env(1) invocations.
Sponsored by:	The FreeBSD Foundation
2016-02-09 20:12:26 +00:00
Glen Barber
a5395435ff Add logic to rotate the package repository, keyed on PKG_VERSION,
and create a 'latest' symlink to the PKG_VERSION repository path.

Suggested by:	des
Sponsored by:	The FreeBSD Foundation
2016-02-09 19:27:38 +00:00
Dag-Erling Smørgrav
ea85204f94 Rename the signpackages target to sign-packages, and include it in the
packages meta-target so 'make packages' now does everything.
2016-02-08 16:34:13 +00:00
Dag-Erling Smørgrav
ee1627c4db Split the packages target into stage-packages and create-packages to make
it possible to roll new packages from an existing build without having to
restage them.
2016-02-08 16:08:13 +00:00
Glen Barber
bbb51924bb MFH
Sponsored by:	The FreeBSD Foundation
2016-02-08 12:16:01 +00:00
Enji Cooper
71b7fa1252 Simplify running the FreeBSD test suite
Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.

`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.

Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.

Document `make check` and `make checkworld` in build(7).

Other minor changes:

- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
  `make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
  not found

The MFC is [partly] contingent on other build related changes being MFCed.

Differential Revision: https://reviews.freebsd.org/D4406
MFC after: 2 months
X-MFC to: stable/10
Relnotes: yes
Reviewed by: bdrewery, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division
2016-02-07 18:40:04 +00:00
Dimitry Andric
a49d8b6ecc Merge ^/head r294961 through r295350. 2016-02-06 14:07:17 +00:00
Glen Barber
848f2374fe Silence a few command lines.
Sponsored by:	The FreeBSD Foundation
2016-02-05 00:43:42 +00:00
Glen Barber
5c52b2f5ca Rework the way <packagename>.ucl files are generated.
One of the major pain points with how this was implemented
is the requirement of in-tree, hard-coded <name>.ucl, as
well as <name>-<suffix>.ucl where <suffix> can be lib32,
profile, development, debug, or any combination of the four.

This created significant overhead when adding new packages
and any of the files in any of the combinations were missing.

Instead of test(1)-ing if the <packagename>.ucl file exists,
hand off to a script to figure out what the final ucl file
name should be before invoking pkg(8).

The default behavior is 'template.ucl' is used as a fallback.

This affects only the userland packages, as the kernel code
is already smart enough to handle these variations.

Sponsored by:	The FreeBSD Foundation
2016-02-04 15:32:03 +00:00
Bryan Drewery
db9ba9ee1b Deduplicate distrib-dirs logic from r289086 in distribution.
This does it correctly this time, rather than the incorrect version in
r295167.

Sponsored by:	EMC / Isilon Storage Division
2016-02-03 20:24:21 +00:00
Glen Barber
3bea1c132c MFH
Sponsored by:	The FreeBSD Foundation
2016-02-03 02:02:01 +00:00
Conrad Meyer
4be4b11f30 Revert r295167 at bdrewery's request
$ svn merge -c -295167 .

JHB reports Navdeep reports that it breaks distribution and etcupdate.

Approved by:	bdrewery
2016-02-03 01:40:07 +00:00
Glen Barber
221b349912 MFH
Sponsored by:	The FreeBSD Foundation
2016-02-02 22:27:48 +00:00
Bryan Drewery
fb872bbc68 Deduplicate distrib-dirs logic from r289086 in distribution.
Sponsored by:	EMC / Isilon Storage Division
2016-02-02 21:11:23 +00:00
Bryan Drewery
7cd7d12629 Adjust install .WAITs for lib/ and etc/ to allow parallelization more.
Only 'installworld' needs to be protected and only when not using
-DNO_ROOT, which implies not installing to / and not needing the
lib dependency protections.

Sponsored by:	EMC / Isilon Storage Division
2016-02-02 20:50:09 +00:00
Glen Barber
1793d3395d Wrap long lines after recent commits.
Sponsored by:	The FreeBSD Foundation
2016-01-29 21:42:27 +00:00
Glen Barber
35854cea8b Make sure 'pkg config ABI' uses ABI_FILE for the target.
Sponsored by:	The FreeBSD Foundation
2016-01-29 21:39:44 +00:00
Glen Barber
d9b0aafdb6 Reorder execution in the 'packages' target so both userland
and kernel are staged before invoking the 'create-<foo>-packages'
targets.

Include PKG_VERSION value in the 'create-{world,kernel}-packages'
targets so the value is not redefined when packaging the kernel,
which otherwise results in inconsistent and confusing package
version results.

Sponsored by:	The FreeBSD Foundation
2016-01-29 17:05:30 +00:00
Glen Barber
d7a8d3868c Avoid excessively invoking 'pkg bootstrap' if already done.
Sponsored by:	The FreeBSD Foundation
2016-01-28 23:59:34 +00:00
Glen Barber
f61b169160 Fix naming for multiple kernels when building/packaging more than one.
Sponsored by:	The FreeBSD Foundation
2016-01-27 21:24:45 +00:00
Glen Barber
3d98f473dc Add 'stagekernel' target, which invokes 'distributekernel'
with -DNO_ROOT to create the METALOG mtree(8) file.

Separate the default STAGEDIR for world (WSTAGEDIR) and kernel
(KSTAGEDIR).

Fix the 'create-kernel-packages' target to work properly.

Evaluate if 'kernel' is set when invoking mtree-to-plist.awk,
which splits the kernel and kernel.debug into separate plist
files.

Fix METALOG creation when building/packaging multiple kernels.

Sponsored by:	The FreeBSD Foundation
2016-01-27 21:10:03 +00:00
Glen Barber
d46f04328b Add a 'signpackages' target, which creates the pkg repository
metadata files and uses the path of PKGSIGNKEY for signing,
if set.

Sponsored by:	The FreeBSD Foundation
2016-01-25 16:18:08 +00:00
Glen Barber
f9371dca60 Create a _pkgbootstrap target, and remove the env(1) prefix
to pkg(8) commands.

Move the resulting packages outside STAGEDIR to minimize
pollution.

When invoking 'pkg create', include the ABI in the REPODIR
path so the correct hierarchy is created for cross-builds.

Move the STAGEDIR and REPODIR declarations above the targets
that use them to keep things cleaner, and move the packages
target.

Include '-o ABIFILE=DESTDIR/bin/sh' in pkg(8) invocations in
the create-kernel-packages target.

Sponsored by:	The FreeBSD Foundation
2016-01-22 18:06:09 +00:00
Glen Barber
bc3f9191c7 Ensure pkg(8) is bootstrapped, and avoid prompting for confirmation
to install, in case the 'packages' target is run non-interactively.

Sponsored by:	The FreeBSD Foundation
2016-01-21 21:42:06 +00:00
Dimitry Andric
ea2c42d8d3 Merge ^/head r293686 through r293849. 2016-01-13 20:16:53 +00:00
Glen Barber
e60680024a MFH
Sponsored by:	The FreeBSD Foundation
2016-01-12 01:23:45 +00:00
Bryan Drewery
57ce33acfe Correct a comment.
Submitted by:	jhb
2016-01-11 20:27:05 +00:00
Dimitry Andric
89d3f0ea4e Merge ^/head r293430 through r293685. 2016-01-11 19:36:44 +00:00
Bryan Drewery
1f17dba482 Chase r292622: Update path to ioctl.c for incremental build hack.
Sponsored by:	EMC / Isilon Storage Division
2016-01-08 22:59:49 +00:00
Bryan Drewery
8dc7b23a02 Fix upgrading from OSVERSION 1000002-1000032 after r288829.
r288829 states that lex requires the latest m4, but was not always building it.
Move lex to the same logic as m4 since they are closely tied now.

MFC after:	3 days
Sponsored by:	EMC / Isilon Storage Division
Reported by:	Slawa Olhovchenkov <slw@zxy.spb.ru>
2016-01-08 21:07:34 +00:00
Dimitry Andric
b229c1a0b0 Merge ^/head r293280 through r293429. 2016-01-08 17:42:14 +00:00
Glen Barber
0c9da521fa MFH
Sponsored by:	The FreeBSD Foundation
2016-01-07 17:23:43 +00:00
Gleb Smirnoff
b46e917554 Provide knob NO_INSTALLEXTRAKERNELS. If defined, extra kernels in KERNCONF
won't be installed, only the first one would.
2016-01-06 21:58:45 +00:00
Dimitry Andric
5dc6cc42f4 Merge ^/head r293175 through r293279. 2016-01-06 21:31:07 +00:00
Glen Barber
1a5c9277ca MFH
Sponsored by:	The FreeBSD Foundation
2016-01-05 21:12:49 +00:00
Glen Barber
8834318685 Merge ^/projects/release-install-debug:
- Rework MANIFEST generation and parsing via bsdinstall(8).
- Allow selecting debugging distribution sets during install.
- Rework bsdinstall(8) to fetch remote debug distribution sets
  when they are not available on the local install medium.
- Allow selecting additional non-GENERIC kernels during install.
  At present, GENERIC is still required, and installed by default.

Tested with:	head@r293203
Sponsored by:	The FreeBSD Foundation
2016-01-05 21:05:17 +00:00
Glen Barber
c60009ccf4 In 'create-kernel-packages', do not exit with an error if NO_ROOT is
not set.  The 'stageworld' target is invoked with -DNO_ROOT, so the
metalog file(s) will be created regardless.

This matches the behavior of 'create-world-packages'.

Sponsored by:	The FreeBSD Foundation
2016-01-05 18:05:31 +00:00
Glen Barber
51daa9fe00 Do not implicitly invoke 'buildworld' as part of the 'packages' target.
This retains the original behavior of release-related targets, which
assume 'buildworld' and 'buildkernel' have already happened.

Sponsored by:	The FreeBSD Foundation
2016-01-05 15:50:50 +00:00
Glen Barber
b626f5a73a MFH r289384-r293170
Sponsored by:	The FreeBSD Foundation
2016-01-04 19:19:48 +00:00
Dimitry Andric
34cdd77646 First part of updating llvm/clang build glue: getting llvm-tblgen,
clang-tblgen and clang itself built.
2015-12-31 14:39:45 +00:00
Sean Bruno
7144d5cbc5 Fixup native-xtools target for poudriere cross build jails after svn
r291955 by excluding all debug files.

Reported by:	swills

Reviewed by:	emaste
2015-12-23 21:55:54 +00:00
Enji Cooper
4a2a740e17 Always expose LOCALBASE, not just when CROSS_TOOLCHAIN is defined
Instead of using which(1) to look for doxygen, look for it in <LOCALBASE>/bin .
$PATH gets mangled by make buildenv, etc so it's better to just be explicit
about the path if someone uses that for instance.

Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff)
Reviewed by: emaste, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division
2015-12-20 05:52:03 +00:00
Bryan Drewery
37ded2a72c Follow-up r290423: Don't use CSH for buildenv shell.
It does not properly import PATH; the PATH is reset by included profile
files on startup which breaks the biggest feature of buildenv (using
sysrooted cc from WORLDTMP)

Spotted by:	smh, kib
Sponsored by:	EMC / Isilon Storage Division
2015-12-14 23:25:31 +00:00
Andrew Turner
c492eb76dd Split KNOWN_ARCHES so we have one per line to simplify merging future
architectures.
2015-12-14 09:52:09 +00:00
Ruslan Bukin
1fdcc5e5c0 Start support for the RISC-V 64-bit architecture developed by UC Berkeley.
RISC-V is a new ISA designed to support computer research and education, and
is now become a standard open architecture for industry implementations.

This is a minimal set of changes required to run 'make kernel-toolchain'
using external (GNU) toolchain.

The FreeBSD/RISC-V project home: https://wiki.freebsd.org/riscv.

Reviewed by:	andrew, bdrewery, emaste, imp
Sponsored by:	DARPA, AFRL
Sponsored by:	HEIF5
Differential Revision:	https://reviews.freebsd.org/D4445
2015-12-11 22:55:23 +00:00
Bryan Drewery
9229b1797a Fix some makeman issues.
- Don't bother looking up REVISION/BRANCH/etc from release/, or the
  CPUTYPE check, as these are not used for makeman and wastes time.  The also
  invokes auto.obj.mk after I reverted auto.obj.mk ignoring -V in r291312.
- Don't modify CC or PATH when WITH_CCACHE_BUILD or WITH_META_MODE is enabled
  as it leads to bsd.compiler.mk errors.

Sponsored by:	EMC / Isilon Storage Division
2015-12-08 17:09:17 +00:00
Bryan Drewery
1c2a618bbf Fix spelling of internal hack.
Reported by:	ngie
2015-12-07 18:45:55 +00:00
Bryan Drewery
03173d2ff0 bsd.subdir.mk: Only recurse on called targets, rather than dependencies.
This is to fix 'make all' causing it to recurse on both 'all' and 'buildconfig'
due to 'buildconfig' being in ALL_SUBDIR_TARGETS and being a dependency of
'all'.

This now adds all of the '*includes', '*files' targets as subdir targets,
allowing them to recurse.

This also removes the need for some 'realinstall' hacks in bsd.subdir.mk since
it no longer recurses; only 'install' will recurse and call the proper
'beforeinstall', 'realinstall', and 'afterinstall' in each sub-directory.

This fixes 'make includes' and 'make files' to not be a rerolled ${MAKE}
sub-shell but to rather just recurse on 'inclues' and 'files'.  This avoids
various issues such as the one fixed in r289462.  As such revert Makefile.inc1
back to using 'includes' which avoids an extra tree walk and parallelizes
the includes phases better.

Makefile.inc1 includes a guard so that 'make all' will not use SUBDIR_PARALLEL,
added in r289438.  This is so users do not get a probably broken build if they
run 'make all' from the top-level.  Before the change in this commit, the
workaround for 'make everything' was 'par-all' which would depend on 'all' and
cause a proper parallel recursion.  Now that will not work so a new
_PARALLEL_SUBUDIR_OK is used to allow it.

This is still part of an effort to combine bsd.(files|incs|confs).mk and move
some of its logic out of bsd.subdir.mk, as attempted in r289282 and reverted in
r289331.  This commit fixes the problems found there which was mostly double
recursing during 'includes' which would recurse on itself and 'buildincludes'
and 'installincludes', all in parallel.  The logic is still in bsd.subdir.mk
for now.

I've been cautious about this commit but have experienced no breakage on the
tree except for the 'par-all' case which was already a hack.  If something foo
is depending on something bar that should recurse, it is very likely that the
foo target is being recursed on already meaning that bar will still effectively
recurse once sub-directories call foo.

Discussed on:	arch@
MFC after:	never
Sponsored by:	EMC / Isilon Storage Division
2015-12-02 01:50:22 +00:00
Bryan Drewery
6fa9702035 Add NO_INSTALLKERNEL to undo the assumption that the first KERNCONF will be
installed as "kernel".  This is relevant for packaging of the kernel when
not wanting a default "kernel.txz".

Submitted by:	Russell Cattelan <cattelan@thebarn.com>
MFC after:	2 weeks
Obtained from:	OneFS
Sponsored by:	EMC / Isilon Storage Division
2015-12-01 20:24:58 +00:00
Bryan Drewery
0a63717cfa Allow storing package(world|kernel) tarballs into a different location at PACKAGEDIR.
Submitted by:	Russell Cattelan <cattelan@thebarn.com>
Discussed with:	gjb
MFC after:	2 weeks
Obtained from:	OneFS
Sponsored by:	EMC / Isilon Storage Division
2015-12-01 19:58:08 +00:00
Bryan Drewery
b67490a3b0 Fix errors being ignored in many phases of the build since the bmake integration.
Say it with me, "I will not chain commands with && in Makefiles"

This was originally fixed and explained quite well by bde@ in r36074.  The
initial bmake integration caused 'set -e' to stop being used which lead to
r252419.  Later 'set -e' expectations were fixed with bmake in r254980.

Because of the && here, errors would be ignored when building in parallel and
a dependency failed.  Such as bootstrap-tools since it builds everything in
parallel.  If any tool failed in obj/depend/all, it would just ignore the error
and continue to build.  This later would result in cascaded errors that only
confused the real issue.  This could also cause commands after the failed
command to still execute, leading to more confusion.

This should be fine if the command is in a sub-shell such as: (cmd1 && cmd2)

This reverts r252419.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-12-01 19:00:43 +00:00
John Baldwin
7f911abe54 Add support to libkvm for reading vmcores from other architectures.
- Add a kvaddr_type to represent kernel virtual addresses instead of
  unsigned long.
- Add a struct kvm_nlist which is a stripped down version of struct nlist
  that uses kvaddr_t for n_value.
- Add a kvm_native() routine that returns true if an open kvm descriptor
  is for a native kernel and memory image.
- Add a kvm_open2() function similar to kvm_openfiles().  It drops the
  unused 'swapfile' argument and adds a new function pointer argument for
  a symbol resolving function.  Native kernels still use _fdnlist() from
  libc to resolve symbols if a resolver function is not supplied, but cross
  kernels require a resolver.
- Add a kvm_nlist2() function similar to kvm_nlist() except that it uses
  struct kvm_nlist instead of struct nlist.
- Add a kvm_read2() function similar to kvm_read() except that it uses
  kvaddr_t instead of unsigned long for the kernel virtual address.
- Add a new kvm_arch switch of routines needed by a vmcore backend.
  Each backend is responsible for implementing kvm_read2() for a given
  vmcore format.
- Use libelf to read headers from ELF kernels and cores (except for
  powerpc cores).
- Add internal helper routines for the common page offset hash table used
  by the minidump backends.
- Port all of the existing kvm backends to implement a kvm_arch switch and
  to be cross-friendly by using private constants instead of ones that
  vary by platform (e.g. PAGE_SIZE).  Static assertions are present when
  a given backend is compiled natively to ensure the private constants
  match the real ones.
- Enable all of the existing vmcore backends on all platforms.  This means
  that libkvm on any platform should be able to perform KVA translation
  and read data from a vmcore of any platform.

Tested on:	amd64, i386, sparc64 (marius)
Differential Revision:	https://reviews.freebsd.org/D3341
2015-11-27 18:58:26 +00:00
Bryan Drewery
20f680588a Remove unneeded libmd from bootstrap-tools (reverting r246784).
The bootstrap-tools are supposed to be host tools, which in most cases, use
host headers and libraries.  As such, directly including the src tree's headers
for libmd here causes the need to link libmd in since it will be built with
the new symbols (which /usr/lib/libmd.so) won't have unless it is new enough.
During the target build in buildworld the target headers are staged into
WORLDTMP and used via --sysroot, allowing the target xinstall to be built with
the new/target libmd.

The .PATH here was also not doing anything since xinstall does not use libmd
source files.

Sponsored by:	EMC / Isilon Storage Division
MFC after:	2 weeks
2015-11-18 21:37:46 +00:00
Brad Davis
2cc77b79b8 Don't match on "version" when found as a value.
Approved by:	bapt
2015-11-17 21:10:15 +00:00
Bryan Drewery
f2307a2a25 Avoid setting schg in the objtree for lib32 build.
Reported by:	kib
Sponsored by:	EMC / Isilon Storage Division
MFC after:	1 week
2015-11-14 01:45:37 +00:00
Bryan Drewery
ddd945c7b5 Fix 'make cleanworld' to respect TARGET/TARGET_ARCH for cross-build objtrees.
This simplifies the logic to always try removing the objdir if it exists
and to fallback on a 'cleandir' if no objdir exists.  The reasoning for
this is to avoid rm -rf src/* (r126024)

Sponsored by:	EMC / Isilon Storage Division
MFC after:	2 weeks
2015-11-09 23:36:57 +00:00
Bryan Drewery
2753ba1ab6 Move 'make cleandir' from Makefile to Makefile.inc1.
This leads the way for fixing cross-build cleanup, and eventually replacing
'cleandir' with it during the build.

Sponsored by:	EMC / Isilon Storage Division
MFC after:	2 weeks
2015-11-09 23:36:46 +00:00
Bryan Drewery
9160419c7a Add built-in ccache build support via WITH_CCACHE_BUILD option.
ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build.  This is explained in
more detail upstream [1] [2].  It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times.  When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'.  Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports.  This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead.  ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss.  For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct.  Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it.  Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings.  GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual.  Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization.  There is also a memcached backend available [5].  Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout.  A
possible working layout is as follows:
  Source: /some/prefix/src1
  Source: /some/prefix/src2
  Source: /some/prefix/src3
  Objdir: /some/prefix/obj
  Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative.  Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
  WITNESS
  Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
  DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
        The arc was fully populated with src tree files and ccache objects.
  RAM: 76GiB
  CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
       2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
  x buildworld-before
  + buildworld-ccache-empty
  * buildworld-ccache-full
  % buildworld-ccache-full-fastdep
  # buildworld-fastdep
  +-------------------------------------------------------------------------------+
  |%            *                               #                                +|
  |%            *                               #                                +|
  |%            *                               #             xxx                +|
  |                                                           |A                  |
  |                                                                              A|
  |             A                                                                 |
  |A                                                                              |
  |                                             A                                 |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3       3744.13       3794.31       3752.25     3763.5633     26.935139
  +   3          4519       4525.04       4520.73       4521.59     3.1104823
  Difference at 95.0% confidence
          758.027 +/- 43.4565
          20.1412% +/- 1.15466%
          (Student's t, pooled s = 19.1726)
  *   3       1823.08        1827.2       1825.62        1825.3     2.0785572
  Difference at 95.0% confidence
          -1938.26 +/- 43.298
          -51.5007% +/- 1.15045%
          (Student's t, pooled s = 19.1026)
  %   3       1266.96       1279.37       1270.47     1272.2667     6.3971113
  Difference at 95.0% confidence
          -2491.3 +/- 44.3704
          -66.1952% +/- 1.17895%
          (Student's t, pooled s = 19.5758)
  #   3       3153.34       3155.16        3154.2     3154.2333    0.91045776
  Difference at 95.0% confidence
          -609.33 +/- 43.1943
          -16.1902% +/- 1.1477%
          (Student's t, pooled s = 19.0569)

buildkernel:
  x buildkernel-before
  + buildkernel-ccache-empty
  * buildkernel-ccache-empty-fastdep
  % buildkernel-ccache-full
  # buildkernel-ccache-full-fastdep
  @ buildkernel-fastdep
  +-------------------------------------------------------------------------------+
  |#                        @   %                  *                              |
  |#                        @   %                  *     x                      + |
  |#                        @   %                  *     xx                     ++|
  |                                                      MA                       |
  |                                                                             MA|
  |                                                A                              |
  |                             A                                                 |
  |A                                                                              |
  |                         A                                                     |
  +-------------------------------------------------------------------------------+
      N           Min           Max        Median           Avg        Stddev
  x   3        571.57        573.94        571.79     572.43333     1.3094401
  +   3        727.97        731.91        728.06     729.31333     2.2492295
  Difference at 95.0% confidence
          156.88 +/- 4.17129
          27.4058% +/- 0.728695%
          (Student's t, pooled s = 1.84034)
  *   3         527.1        528.29        528.08     527.82333    0.63516402
  Difference at 95.0% confidence
          -44.61 +/- 2.33254
          -7.79305% +/- 0.407478%
          (Student's t, pooled s = 1.02909)
  %   3         400.4        401.05        400.62        400.69     0.3306055
  Difference at 95.0% confidence
          -171.743 +/- 2.16453
          -30.0023% +/- 0.378128%
          (Student's t, pooled s = 0.954969)
  #   3        201.94        203.34        202.28        202.52    0.73020545
  Difference at 95.0% confidence
          -369.913 +/- 2.40293
          -64.6212% +/- 0.419774%
          (Student's t, pooled s = 1.06015)
  @   3        369.12        370.57         369.3     369.66333    0.79033748
  Difference at 95.0% confidence
          -202.77 +/- 2.45131
          -35.4225% +/- 0.428227%
          (Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR:		182944 [4]
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
Relnotes:	yes
2015-11-08 00:50:18 +00:00
Bryan Drewery
a55d4ddd20 Add a hack to workaround ZSH as BUILDENV_SHELL breaking CPUTYPE.
ZSH considers CPUTYPE a magic variable that will be the output of 'uname -m'
even if already set in environment when starting up.

The CPUTYPE?= check in Makefile.inc1 and supporting overriding CPUTYPE
manually in the buildenv shell make automatic workarounds too tricky
here.  ZSH should really respect variables set in the environment before
trashing them.

X-MFC-With:	r290423
MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-11-08 00:50:06 +00:00
Baptiste Daroussin
3e4f384ed2 Merge from head r290483 2015-11-07 11:02:33 +00:00
Bryan Drewery
208f108362 Allow 'make buildenv' to work anywhere in the src tree.
Sponsored by:	EMC / Isilon Storage Division
2015-11-05 22:28:31 +00:00
Bryan Drewery
e4fa4fafab Allow 'make buildenv' to default to the caller's shell by using SHELL.
Also pass BUILDENV=1 into the sub-shell to allow modifying PS1 in .profile such
as:
  if [ -n "${BUILDENV}" ]; then
          PS1="(buildenv) ${PS1}"
  fi

SHELL defaults to 'sh' in share/mk/sys.mk, but is typically passed down by
the shell invoking make as well.  Rather than forcing all 'buildenv' users
to use plain /bin/sh, let them use their favorite shell.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
Discussed with:	imp
2015-11-05 22:09:00 +00:00
Warner Losh
e5ff3aad8d Fix CC being wrong during install* targets.
Move CROSS_TOOLS stuff to top of file (before bsd.compiler.mk) so that
decisions made by bsd.compiler.mk can properly affect the defaults in
src.opts.mk. Move that to after bsd.compiler.mk. Add a comment about
why we include bsd.compiler.mk here despite the fact that src.opts.mk
currently does too. Also remove bsd.arch.inc.mk that's been OBE.

Differential Revision: https://reviews.freebsd.org/D4087
2015-11-05 21:48:12 +00:00
Craig Rodrigues
a36eca08bb Rename libohash to libopenbsd.
libopenbsd is an internal library which
to bring in compatibility stuff from OpenBSD.
This will allow us to bring in more
OpenBSD utilities into the FreeBSD base system.

We similarly use libnetbsd for bringing in stuff from NetBSD.

Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D4078
2015-11-04 07:20:55 +00:00
Baptiste Daroussin
3c3feed41a Merge from head 2015-11-01 21:17:38 +00:00
Bryan Drewery
6aa2fe8969 Add a note about DEPFLAGS which is currently a hack around not properly passing
CXXFLAGS to sub-makes.

The bad passing also causes bsd.dep.mk's logic to selectively pull only some
flags from C[XX]FLAGS to not apply which can be seen with '-L' being passed to
mkdep when using an external compiler.
2015-10-26 19:28:20 +00:00
Bryan Drewery
c9ec6256cb native-xtools: Replace common path with NXBDESTDIR.
Also combine some mkdir calls.

Sponsored by:	EMC / Isilon Storage Division
2015-10-24 01:39:31 +00:00
Bryan Drewery
5eedd0c0e4 native-xtools: Fix build with WITH_DEBUG_FILES.
Sponsored by:	EMC / Isilon Storage Division
2015-10-23 23:07:45 +00:00
Bryan Drewery
68847f1449 Parallelize build-tools.
MFC after:	3 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-10-22 05:01:14 +00:00
Ed Maste
8f9403e952 Warn that NOCLEAN is deprecated - use NO_CLEAN instead
NO_CLEAN has been the correct spelling for over a decade.
2015-10-22 01:32:11 +00:00
Bryan Drewery
7f1636b75e Let SUBDIR_OVERRIDE with 'make buildworld' be more useful.
Now it can be used to effectively "build in a subdir".  It will use the
'cross-tools', 'libraries', and 'includes' phases of 'buildworld' to properly
setup a WORLDTMP to use.  Then it will build 'everything' only in the
listed SUBDIR_OVERRIDE directories.  It is still required to list custom
library directories in LOCAL_LIB_DIRS if SUBDIR_OVERRIDE is something
that contains libraries outside of the normal area (such as
SUBDIR_OVERRIDE=contrib/ofed needing LOCAL_LIB_DIRS=contrib/ofed/usr.lib)

Without these changes, SUBDIR_OVERRIDE with buildworld was broken or hit
obscure failures due to missing libraries, includes, or cross compiler.

SUBDIR_OVERRIDE with 'make <target that is not buildworld>' will continue to
work as it did before although its usefulness is questionable.

With a fully populated WORLDTMP, building with a SUBDIR_OVERRIDE with
-DNO_CLEAN only takes a few minutes to start building the target
directories.  This is still much better than building unneeded things via
'everything' when testing small subset changes.  A BUILDFAST or
SKIPWORLDTMP might make sense for this as well.

- Add in '_worldtmp' as we still need to create WORLDTMP as later targets,
  such as '_libraries' and '_includes' use it.  This probably was avoiding
  calling '_worldtmp' to not remove WORLDTMP for debugging purposes, but
  -DNO_CLEAN can be used for that.

- '_legacy' must be included since '_build-tools' uses -legacy.
  The SUBDIR_OVERRIDE change came in r95509, while -legacy being part
  of build-tools came in r113136.

- 'bootstrap-tools' is still skipped as this feature is not for
   upgrades.

- Fix buildworld combined with SUBDIR_OVERRIDE not installing all includes.

  The original change for SUBDIR_OVERRIDE in r95509 kept '_includes'
  and '_libraries' as building everything possible as the SUBDIR_OVERRIDE
  could need anything from them.  However in r96462 the real 'includes'
  target was changed from manual sub-makes to just recursing 'includes'
  on SUBDIR, thus not all includes have been installed into WORLDTMP since then
  when combined with 'buildworld'.

  This is not done unless calling 'make buildworld' as it would be
  unexpected to have it go into all directories when doing 'make
  SUBDIR_OVERRIDE=mydir includes'.

- Also need to build the cross-compiler so it is used with --sysroot.
  If this is burdensome then telling the build to use the local compiler
  as an external compiler (thus using a proper --sysroot to WORLDTMP) is
  possible by setting CC=/usr/bin/cc, CXX=/usr/bin/c++, etc.

- Don't build the lib32 distribution with SUBDIR_OVERRIDE in buildworld
  since it won't contain anything related to SUBDIR_OVERRIDE.  Testing
  of the lib32 build can be done with 'make build32'.

- Document these changes in build.7

Sponsored by:	EMC / Isilon Storage Division
MFC after:	2 weeks
2015-10-22 00:07:48 +00:00
Bryan Drewery
e0499c1184 Fix my change in r289435 causing 'etc' to be added to SUBDIR when using
SUBDIR_OVERRIDE.

MFC after:	2 weeks
X-MFC-With:	r289435
Sponsored by:	EMC / Isilon Storage Division
2015-10-21 18:58:46 +00:00
Bryan Drewery
d82fd861a1 Add some missing '+', .MAKE, and .PHONY modifiers.
Some of these targets were lacking both .MAKE and a '+'.  Others were just
inconsistent.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-10-21 05:33:48 +00:00
Julio Merino
74c52c6526 Handle lib32 files during delete-old* when MK_LIB32=no.
Extend OptionalObsoleteFiles.inc to delete all lib32 files when MK_LIB32 is
set to no on a system that previously had lib32 libraries installed.

Also, to prevent "make delete-old-dirs" from always deleting lib32 directories
after an installworld, move the lib32 subtree to its own mtree file that only
gets applied when MK_LIB32=yes.

Test: Ran "make delete-old" and "make delete-old-libs" on a system that never
had MK_LIB32 enabled, and on a system where MK_LIB32 was enabled and later
disabled.  Did this both on amd64 and powerpc64.

Test: Ran "make tinderbox" without errors.

Reviewed by:	emaste
Differential Revision:	https://reviews.freebsd.org/D3923
2015-10-20 20:35:34 +00:00
Baptiste Daroussin
031c294c1d Merge from head 2015-10-19 11:51:10 +00:00
Bryan Drewery
b02f187c44 Remove unneeded MK_CTF=no when MK_CDDL=no.
This has been handled since r228158 made MK_CTF dependent on MK_CDDL
in share/mk/bsd.opts.mk.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-10-17 18:59:14 +00:00
Bryan Drewery
4c339735da Fix wrong PATH being set for world 'includes' stage after r289438.
The 'includes' target is currently a pseudo target in bsd.subdir.mk that
does 'cd ${.CURDIR} && ${MAKE} buildincludes && ${MAKE} installincludes',
versus all over targets that just recurse.

In Makefile.inc1 the older duplicated bsd.subdir.mk logic for calling
'includes' was being executed in each subdir directly, meaning 'cd lib && make
includes' became 'cd lib && make buildincludes && make installincludes'.  Now
that the bsd.subdir.mk logic is used it is calling 'make buildincludes && make
installincludes' from the top-level which pulls in the PATH=<default path>
from /Makefile.

The sub-make logic for 'includes' in bsd.subdir.mk was attempted to be removed
in r289282 but turned out to be wrong.  I have a working version now but
it is not yet ready for commit.  So for now in Makefile.inc1 split out
'includes' to 'buildincludes' and 'installincludes' which will avoid the
problem.

MFC after:	2 weeks
X-MFC-With:	r289438
Sponsored by:	EMC / Isilon Storage Division
2015-10-17 18:22:18 +00:00
Bryan Drewery
53c0e6d54b For 'buildenvvars' show any .exported variables as well to cover recent
exporting of OSRELDATE and VERSION.  These already do export to 'buildenv'
fine.

Sponsored by:	EMC / Isilon Storage Division
2015-10-17 05:57:29 +00:00
Bryan Drewery
a378087d24 Always export VERSION to the environment to avoid looking it up again in
sub-makes.

Some of the world phases that used plain '${MAKE} -f Makefile.inc1' were not
passing this variable along which caused them to look it up again.  By
using bmake's .export we can remove it from all of the other environment
lines.

Add a comment about the usage for VERSION for ctfmerge.

Sponsored by:	EMC / Isilon Storage Division
2015-10-17 05:55:45 +00:00
Bryan Drewery
470c2b3af1 Export OSRELDATE so sub-makes don't look it up again.
We pass BOOTSTRAPPING=${OSRELDATE} to some of the sub-makes.  Rather than
chase every ${MAKE} invokation, just export it as bmake lets us.

Sponsored by:	EMC / Isilon Storage Division
2015-10-17 04:03:53 +00:00
Bryan Drewery
fb84a99e0f Rework the world subdir build targets to use the standard SUBDIR_PARALLEL mechanism.
Back in r30113, the 'par-*' targets were added to parallelize portions of
the build in a very similar fashion as the SUBDIR_PARALLEL feature used in
r263778.  Calling a target without 'par-' (for 'parallel') resulted in the
standard bsd.subdir.mk handling without parallelization.  Given we have
SUBDIR_PARALLEL now there is no reason to duplicate the handling here.

In build logs this will result in the ${dir}.${target}__D targets now showing
as the normal ${target}_subdir_${dir} targets.

I audited all of the uses of Makefile.inc1 and Makefile's targets that use
bsd.subdir.mk and found that all but 'all' and 'install' were fine to use
as always parallel.
  - For 'install' (from installworld -j) the ordering of lib/ and libexec/
    before the rest of the system (described in r289433), and etc/ being last
    (described in r289435), is all that matters.  So now a .WAIT is added in
    the proper places when invoking any 'install*' target.  A parallel
    installworld does work and took 46% of the time a non-parallel
    install would take on my system with -j15 to ZFS.
  - For 'all' I left the default handling for this to not run in parallel.  A
    'par-all' target is still used by the 'everything' stage of buildworld
    to continue building in parallel as it already has been.  This works
    because most of the dependencies are handled by the early bootstrap
    phases as well as 'libraries' and 'includes' phases.  This lets
    all of the SUBDIR build in parallel fine, such as bin/ and lib/.  This
    will not work if the user invokes 'all' though as we have dependencies
    spread all over the system with no way to depend between them (except
    for the dirdeps feature in the META_MODE build).  Calling 'make all'
    from the top-level is still useful at least when using SUBDIR_OVERRIDE.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-10-17 03:51:50 +00:00
Bryan Drewery
23ffbc1011 Fix adding manpages installed by LOCAL_DIRS to whatis file.
The ordering of 'etc' in the install has a long history dating back to the
first time it was realized it needed to be "last" in r4486.  That commit
still left it before LOCAL_DIRS though.  By having it before LOCAL_DIRS
any manpages they install were not being added to the whatis database in the
install image.  They would likely show up in the file after a periodic
rebuild of the file though.

Currently the whatis file is built by an 'afterinstall' hook in etc/Makefile
that calls share/man's 'makedb' target.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-10-16 23:53:37 +00:00
Bryan Drewery
c38230eb54 Remove lockf as an ITOOL.
It was added in r152006 to handle serializing access of info/dir when
installing INFO files.  We no longer support INFO files since r276551
though.

Sponsored by:	EMC / Isilon Storage Division
2015-10-16 22:41:31 +00:00
Bryan Drewery
7e81b832a3 Correct a bitrotted comment about installworld order requirements.
The case of make(1) using a new /bin/sh issue was fixed in r173219 when ITOOLS
was introduced.

There are still issues with mid-install errors leaving a system unusable that
are currently non-trivial to solve.  The safest ordering requires installing
rtld, libc and libthr (in that order) before anything else.  We don't do that
now though.  Much improvement is needed here still.

Discussed with:	kip and kan (rtld/library ordering)
MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-10-16 21:09:15 +00:00
Bryan Drewery
35511448cd Remove .MAKE from targets that do more than just run sub-makes, such as
calling rm or mtree.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-10-16 20:23:04 +00:00
Bryan Drewery
651e270328 Fix delete-old and check-old-files not removing old debug symbols.
This was handled for libraries in r256842 but for some reason was missed
for files (bsd.prog.mk).

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
Relnotes:	yes
2015-10-16 04:07:27 +00:00
Bryan Drewery
7b9c2f1643 Fix buildworld with clean objdir after r289351.
Sponsored by:	EMC / Isilon Storage Division
2015-10-15 21:21:41 +00:00
Glen Barber
86f589087c Pass .MAKEFLAGS in the 'packages' target to allow parallel
builds.

Pass '-B' when invoking the 'stageworld' target, similar to
what is done for 'installworld'.

Sponsored by:	The FreeBSD Foundation
2015-10-15 19:53:16 +00:00
Glen Barber
324fd1ce05 MFH to r289370
Sponsored by:	The FreeBSD Foundation
2015-10-15 17:36:56 +00:00
Bryan Drewery
ba831a2cb7 Consider top-level targets to be .PHONY as bmake won't build them
otherwise if a file with the same name is found in the directory.

MFC after:	1 week
Sponsored by:	EMC / Isilon Storage Division
2015-10-15 05:32:56 +00:00
Bryan Drewery
33388a1662 Add temporary workaround for .MAKE being applied to _worldtmp, since r251750.
This was causing files to be removed from the objdir when -n was used.
_worldtmp makes no sub-make calls.

A more comprehensive solution is coming involving fine-grained '+' where
appropriate.

Sponsored by:	EMC / Isilon Storage Division
MFC after:	1 week
2015-10-15 05:26:11 +00:00
Ed Maste
07670b30fa Create /usr/tests *.debug file directory hierarchy
Reviewed by:	bdrewery, ngie
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3896
2015-10-15 01:46:55 +00:00
Baptiste Daroussin
becbad1f6e Merge from head 2015-10-13 19:44:36 +00:00
Baptiste Daroussin
c69900f12f Merge from head 2015-10-10 23:12:36 +00:00
Baptiste Daroussin
5d8d649803 Fix installing config files as non root
Reported by:	adrian
2015-10-10 14:43:46 +00:00
Baptiste Daroussin
9be27fdc00 Merge from head 2015-10-09 22:45:54 +00:00
Baptiste Daroussin
dd985c8f7a Change make distribution so that it now call installconfig in all dirs along
with the current behaviour of calling "distribution" in the etc target.

This allows mergemaster/etcupdate to still work when some configuration will be
moved to be handled in the same directories their source code lives in.
2015-10-09 22:05:31 +00:00
Ian Lepore
89ef7a960a The latest version of lex requires the latest m4 to build, add a dependency
when running the build-tools stage.

The requirement is due to the -P flag used when running m4 from usr.bin/lex
Makefile to generate skel.c.  With the old m4 that fails and the failure is
ignored, resulting in an empty(-ish) skel.c, which leads to later build
failures when the misconfigured new lex tool is run.

This enables building -current (and 10-stable after MFC) on a stable-8
system again.

MFC after:	3 days
2015-10-05 17:45:13 +00:00
Baptiste Daroussin
6875f8e707 Remerge as previous one was broken 2015-10-03 07:22:07 +00:00
Colin Percival
6ae1554a5d Final step of eliminating the "games" distribution: Merge src/games
(or what's left of it, at least) into src/usr.bin.

This change will not be MFCed.

Discussed at:   EuroBSDCon 2014
Committed from: EuroBSDCon 2015
2015-10-02 10:08:11 +00:00
Baptiste Daroussin
5a2b666ce5 Merge from head 2015-10-01 09:36:43 +00:00
Andriy Gapon
9875367b9a remove unused sgsmsg utility (originally imported from opensolaris)
MFC after:	25 days
2015-09-28 12:38:57 +00:00
Bryan Drewery
3cad5c68de META_MODE: Make it easier to keep the bootstrap-tools args in sync.
Sponsored by:	EMC / Isilon Storage Division
2015-09-26 04:22:18 +00:00
Ed Maste
2518edf632 Remove EOL whitespace from Makefile.inc1 2015-09-25 00:30:53 +00:00
Ed Maste
b5512f3b27 Move DTrace tools to cross-tools stage
Bootstrap tools exist for backwards compatibility support. DTrace tools
tools are also needed for cross builds, so belong in cross-tools.

Reviewed by:	imp (earlier), markj
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D2144
2015-09-21 16:07:22 +00:00
Bryan Drewery
26b7188dae Let makeman run 'make showconfig' without hitting the aarch64 error. 2015-09-17 04:48:15 +00:00
Brad Davis
1b87f1f397 Fix grammer in an error message
PR:		202310
Submitted by:	Chris Petrik <chris@bsdjunk.com>
Approved by:	will
2015-09-15 14:24:19 +00:00
Mark Johnston
286bbc3479 Unconditionally build CTF tools in the bootstrap-tools phase of the build.
Stale CTF tools are a frequent source of DTrace issues, and they compile
quickly enough that the increase in build time is negligible.

Reviewed by:	emaste, imp
Differential Revision:	https://reviews.freebsd.org/D3670
2015-09-15 05:01:44 +00:00
Baptiste Daroussin
f94594b37a Finish merging from head, messed up in previous attempt 2015-09-12 12:03:02 +00:00
Baptiste Daroussin
b779ff2b9c Rename kernel-pkgs into create-kernel-packages to be consistent with world 2015-09-12 11:45:10 +00:00
Baptiste Daroussin
b5ff185e19 Merge from head 2015-09-12 11:41:31 +00:00
Gleb Smirnoff
05f05f8bdc The ${BUILDKERNELS:[2..-1]} appears to produce a non zero result for
a one word variable, which is quite unexpected from documentation.
So, to avoid double installation of a single kernel, protect the extra
kernels loop with ${BUILDKERNELS:[#]} > 1 conditional.

Sponsored by:	Netflix
Sponsored by:	Nginx, Inc.
2015-09-02 15:42:14 +00:00
Gleb Smirnoff
f3c7b830e5 Not only build with buildworld, but also install with installworld all
alternative kernels.

Sponsored by:	Netflix
Sponsored by:	Nginx, Inc.
2015-09-01 11:59:12 +00:00
Gleb Smirnoff
6d04a5ad40 When building multiple kernels use [2..-1] to extract !INSTALLKERNEL
from BUILDKERNELS list.  This is more strict, since INSTALLKERNEL by
definition is the first word of BUILDKERNELS list.  The previous
code failed if INSTALLKERNEL is a substring of additional kernel name.

Reviewed by:	gjb
Sponsored by:	Netflix
Sponsored by:	Nginx, Inc.
2015-09-01 11:46:13 +00:00
Baptiste Daroussin
23a32822d2 Merge from HEAD 2015-08-25 20:14:50 +00:00
Baptiste Daroussin
0f3c3059be Remove now unused LIBPRIVATEDIR
Sponsored by:	Gandi.net
2015-08-24 13:46:12 +00:00
Warner Losh
c404eb64d6 Sparc64 is the odd-man out, so form the if that way rather than
listing everybody else.
2015-08-23 23:12:30 +00:00
Ed Maste
084748985d Separate ELFTOOLCHAIN_BOOTSTRAP from BINUTILS_BOOTSTRAP
For most cases they are equivalent, but BINUTILS_BOOTSTRAP is a
BROKEN_OPTION on arm64 as the in-tree GNU binutils do not support it,
so we need a separate internal flag for ELF Tool Chain.

Reviewed by:	andrew, brooks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3381
2015-08-17 10:48:55 +00:00
Warner Losh
39d3a8449c Spell binaries in the customary way.
Submitted by: jhb@
2015-08-13 22:32:42 +00:00
Ed Maste
1bc28ffccd Roll WITHOUT_ELFTOOLCHAIN_TOOLS into WITHOUT_TOOLCHAIN
The option was added only to ease the transition from GNU Binutils to
ELF Tool Chain tools, and that process is now complete (for the viable
replacements). Noting the removal in UPDATING is sufficient as we have
not shipped a release with the option.

Reviewed by:	brooks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D3240
2015-08-13 17:50:47 +00:00
Warner Losh
d7768ad4fa Fix the fixing of the build I broke. rescue/rescue has the right
target, but rescue doesn't.

Pointy hat: imp@
2015-08-12 19:39:11 +00:00
Xin LI
572ec1c57d Fix build. 2015-08-12 19:21:58 +00:00
Warner Losh
9aca5cedfe Document build-tools better. Add rescue back because it builds /bin/sh
which has a build-tools target (see commit for how build-tools and
cross-tools differ).
2015-08-12 19:00:47 +00:00
Warner Losh
1ddcdb76ef Why on earth have we been building rescue as a build tool for the past
12 years? Nothing downstream in the build uses it. Eliminate it as a
build tool.

Reviewed by: emaste@ ("just delete it")
2015-08-12 17:19:52 +00:00
Warner Losh
ad38a9d962 Crunchgen needs to be bootstrapped to pick up the STRIP->STRIPBIN
changes to prevent the 'rescue: not found' errors from happening.
Bump FreeBSD_version to 1100078 since there's been no version bumps
since this change was made. Only people that installed since r284356
really need to do this bootstrapping, but since crunchgen needs to
bootstrap for other reasons, bumping the number was the simplest.
2015-08-12 16:43:15 +00:00
Ed Maste
d99337805f Fix comment describing legacy target and wrap to 80 columns 2015-08-11 17:48:58 +00:00
Ed Maste
8551ff3fba Build libelf and libdwarf in the legacy stage
They need to be built and installed (including headers) prior to the
DTrace CTF tools.

Reviewed by:	imp (as part of a larger change)
Sponsored by:	The FreeBSD Foundation
2015-08-11 15:43:09 +00:00
Warner Losh
f600d35fc1 cmp and cp are used by the kerberos install, so need to be imclided in
ITOOLS. They are tiny enough that I'm not making conditional: the
minuscule savings in disk space isn't worth the obfuscation of
Makefile.inc1.
2015-08-09 18:15:33 +00:00
Baptiste Daroussin
1fd99f14aa libavl is not needed anymore for localedef(1) 2015-08-08 23:05:12 +00:00
Baptiste Daroussin
57df665ac1 Add localedef(1) to bootstrap tools 2015-08-08 00:18:32 +00:00
Ruslan Bukin
b78ee15e9f First cut of DTrace for AArch64.
Reviewed by:	andrew, emaste
Sponsored by:	ARM Limited
Differential Revision:	https://reviews.freebsd.org/D2738
2015-07-01 15:51:11 +00:00
Baptiste Daroussin
af38028d11 Make all shared library a relative symlink
This makes sysroot usable for cross building, it also removes the need for
_SHLIBDIRPREFIX (keeps its definition since picobsd uses it and I have no time
to test it)

Differential Revision:	https://reviews.freebsd.org/D2920
Submitted by:	imp, adrian
Tested by:	adrian
2015-06-27 23:28:56 +00:00
Baptiste Daroussin
0f20a3cdd8 Only build gperf as a bootstrap tool if gcc is going to be built 2015-06-21 22:45:53 +00:00
Baptiste Daroussin
ffe11f412d Remove temporary changes that crept in r284417 when it should not have
Reported by:	antoine, jbeich
2015-06-20 11:11:13 +00:00
Baptiste Daroussin
baec3daef1 Merge from head@274131 2015-06-16 20:01:01 +00:00
Baptiste Daroussin
18b2ee82db Revert r284417 it is not necessary anymore 2015-06-15 19:28:07 +00:00
Baptiste Daroussin
dad2fb7ece Merge from head 2015-06-15 10:56:01 +00:00
Baptiste Daroussin
3e82c72242 Revert r283938 it is not needed anymore and broken
gzip(1) cannot be used as bootstrap tools as it depends on syscalls that only
exists on HEAD
2015-06-03 13:43:04 +00:00
Baptiste Daroussin
0bbe6da5b6 mandoc's makewhatis requires gnuzip to be in the path when deadling with
with compressed manpages.

This should fix spam during installworld: "exec: No such file or directory"

Reported by:	many
2015-06-03 06:02:49 +00:00
Baptiste Daroussin
75e20d65dc Remove libmandoc from bootstrap tools 2015-05-30 19:30:23 +00:00
Baptiste Daroussin
f5e4216e43 Only push libohash once in bootstrap tools 2015-05-30 19:29:19 +00:00
Baptiste Daroussin
8faa40d1a0 Switch to mandoc's version of makewhatis(8), whatis(1), apropos(1) utilities.
This change among other things improve search capabilities over the manpages
allowing fine grain query.

A new build option WITHOUT_MANDOCDB has been added to keep the ancient version
of the database and the tools. The plan is to entirely remove this option before
11.0-RELEASE.

Differential Revision:	https://reviews.freebsd.org/D2603
2015-05-30 17:41:37 +00:00
Baptiste Daroussin
645acf41e3 Do not make libsqlite3 dependency chain only defined for kerberos, soon more
options would need libsqlite3 as well thus depends on this definitions
2015-05-30 16:00:33 +00:00
Glen Barber
37a48d408f MFH: r282615-r283655
Sponsored by:	The FreeBSD Foundation
2015-05-28 17:06:50 +00:00
Baptiste Daroussin
8e9b16e55f Fix build WITHOUT_CDDL by unconditionnaly adding libproc and librtld_db
on supported arches to the prebuild libs
2015-05-26 16:25:28 +00:00
Enji Cooper
2b42dcf7ad _Really_, _Really_ fix buildworld by moving the conditionals down, fixing some
typos, and fixing the dependency when MK_LIBCPLUSPLUS != no

- `:D` operator evaluation is immediate, i.e. like .if defined(..). So r283159
  was in effect a no-op commit.
- Fix dependency in MK_LIBCPLUSPLUS case in two ways:
-- lib/libc++ was the wrong dependency. It should have been libcxxrt.
-- lib/libc++ was missing __L, so again it was depending on the directory, not
   the relevant .PHONY target.

Tested with: make tinderbox (amd64, arm, sparc64) and JFLAG=-j16
In collaboration with: bdrewery, imp, peter
BIG pointyhat to: ngie (for trying to commit things at 6am while staying up all
                        night working on other tasks)
2015-05-21 05:02:08 +00:00
Baptiste Daroussin
f3dba86773 Make kerberos use the same sqlite libraries as other sqlite consumer.
This reduces the number of copy of sqlite we have to just one and easier
tracking version of sqlite

Differential Revision:	https://reviews.freebsd.org/D2443
Reviewed by:	imp, stas, bjk
2015-05-20 18:56:29 +00:00
Enji Cooper
cf13852bab Fix breakage I didn't fully solve in r283151 by depending on the .PHONY
target, not the directory when building libctf for libproc

Reported by: many, Jenkins
Submitted by: rodrigc
2015-05-20 18:41:54 +00:00
Enji Cooper
3f48075933 Unbreak the arm64/sparc64 tinderbox by only compiling lib/libproc and
lib/librtld_db on architectures where they're supported

Reported by: bz, Jenkins
Pointyhat to: bapt
2015-05-20 13:05:33 +00:00
Enji Cooper
ebfd9f66a7 cddl/lib/libctf isn't always a requirement for lib/libproc; its use is dependent
on MK_CTF != "no". Use the other divined value instead of reinventing the wheel
2015-05-20 13:04:00 +00:00
Enji Cooper
5731a3ce7d Add _more_ missing dependencies for lib/libproc to further squash build races
Verified via `cd lib/libproc; make -VLIBADD`

Pointyhat to: ngie
2015-05-20 12:56:40 +00:00
Enji Cooper
d0edea4a0e Articulate all dependencies for lib/libproc to squash build races after r283139
on !arm64 and !sparc64

Pointyhat to: bapt
Sponsored by: EMC / Isilon Storage Division
2015-05-20 12:46:30 +00:00
Enji Cooper
b61e5e4883 Only build sys/boot/usb/tools if MK_USB != no
Sponsored by: EMC / Isilon Storage Division
2015-05-20 09:36:25 +00:00
Baptiste Daroussin
f454747867 Fix buildworld by adding libproc and librtld_db to the _prebuild_libs
Those are needed to build libdtrace
2015-05-20 09:23:37 +00:00
Ed Maste
7f36f2d0e4 Update crunch bootstrapping test for recent fixes
- r277259 crunchide: Correct 64-bit section header offset
- r281674 crunchide: always include both 32- and 64-bit ELF support

With built-in cross-size support we also no longer need a special case
for cross-build crunchide.

Reviewed by:	imp
Sponsored by:	The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D2576
2015-05-19 14:05:15 +00:00
Ed Maste
4d37254910 Remove redundant csu subdir logic
The appropriate subdirectories are handled by lib/csu/Makefile. There's
no need to duplicate this logic in Makefile.inc1 and lib/Makefile.

Reviewed by:	imp
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D2523
2015-05-12 17:53:22 +00:00
Thomas Quinot
55b130956d Unbreak build following rev. 282726
(Makefile.inc1): add dependency of xinstall on libmd to
avoid failure of parallel bootstrap.

(lib/libmd/*.h): do not redefine symbols if already
defined as macros (libcrypt uses the same sources internally,
redefining symbols with a prefix of its own).

Fixes build failures caused by previous change to libmd.

Reported by: ian
Pointy hat to: thomas
2015-05-10 21:21:52 +00:00
Baptiste Daroussin
7757a1b4dc Merge from head 2015-05-03 19:30:11 +00:00
Baptiste Daroussin
363da13804 Replace groff's soelim by soeliminate(1) renamed soelim(1) 2015-05-01 20:08:25 +00:00
Sean Bruno
0484aa593a Setting PATH to anything is useless as a bare command. Its only relevant
if its set in the environement of each command seperately.

Move the PATH setting to the NXBMAKE variable so its picked up to find
the one-off gperf build for the native-xtools target.

Pointed Out by: ngie
2015-04-30 02:08:36 +00:00
Sean Bruno
bf3befadc1 Fix native-xtools breakage when building a gcc enabled target on a clang
enabled host.  Build a one-off gperf and put it in the PATH for the rest
of the target so the ONE call to gperf by the gcc build picks it up and
DTRT.

Reviewed by:	imp
2015-04-30 00:17:32 +00:00
Enji Cooper
9dc634de35 Serialize all of _kerberos5_bootstrap_tools to avoid build failures involving
make bootstrap-tools

On the plus side, this also greatly reduces complexity

MFC after: 1 week
Pointyhat to: ngie
Reported by: Willem Jan Withagen <wjw@digiware.nl>
2015-04-21 10:17:25 +00:00
Ed Maste
2d8d8e3c03 Bump crunch BOOTSTRAPPING for ELF header offset fix in r277557 2015-04-17 16:02:49 +00:00
Warner Losh
c5e8e000fc RELEASEDIR was removed in FreeBSD 9.x, at the same time /boot/loader
stopped using kgzip in the release process. We no longer need to build
kgzip as a cross tool, and tests for RELEASEDIR are obsolete, so
remove both.

Differential Revision: https://reviews.freebsd.org/D2313
2015-04-17 15:55:13 +00:00
Enji Cooper
6c4f929517 Defeat race with MK_KERBEROS == yes introduced with bootstrap-tools
parallelization work done in r279197

- kerberos5/lib/libroken requires kerberos5/tools/make-roken to build
- kerberos5/tools/asn1_compile, kerberos5/tools/slc, and usr.bin/compile_et
  require kerberos5/lib/libroken and kerberos5/lib/libvers

This race is incredibly evident when cross-building sparc64 on
ref10-amd64.freebsd.org

MFC after: 1 week
Pointyhat to: ngie
2015-04-16 21:45:07 +00:00
Warner Losh
008bd7c6f6 People are still getting burned by the byacc upgraded, switch to
always doing byacc until someone figures out the more nuanced version
to switch off of.
2015-04-16 20:50:37 +00:00
Baptiste Daroussin
4bf53d0b46 Merge from HEAD 2015-04-03 23:23:09 +00:00
Andrew Turner
b40d31d0a1 Only use the CROSS_BINUTILS_PREFIX tool if it exists.
Reviewed by:	emaste
Sponsored by:	The FreeBSD Foundation
2015-04-02 17:03:02 +00:00
Dimitry Andric
76e2c537bc Ensure the cross assembler, linker and objcopy are used for the build32
stage, just like for the regular world stage.

Reviewed by:	rodrigc, imp, bapt, emaste
MFC after:	1 week
Differential Revision: https://reviews.freebsd.org/D2187
2015-04-02 06:58:17 +00:00
Craig Rodrigues
5ab58881db Add --sysroot to the compiler flags for clang or gcc external toolchains
Before this fix, --sysroot was only added to the compiler flags if a clang
external toolchain was used.

Reviewed by:   imp
2015-03-26 23:37:03 +00:00
Warner Losh
996d8a0d17 Add some more explanation to the different phases of the build. 2015-03-25 21:59:36 +00:00
Ed Maste
b9a9d9ca9b Force MK_INCLUDES for the legacy stage
As legacy executes "make installincludes" we don't want it to be
disabled by a src.conf setting.

Reviewed by:	imp
MFC after:	1 month
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D2143
2015-03-25 20:57:08 +00:00
Andrew Turner
8daa81674e Start to import support for the AArch64 architecture from ARM. This change
only adds support for kernel-toolchain, however it is expected further
changes to add kernel and userland support will be committed as they are
reviewed.

As our copy of binutils is too old the devel/aarch64-binutils port needs
to be installed to pull in a linker.

To build either TARGET needs to be set to arm64, or TARGET_ARCH set to
aarch64. The latter is set so uname -p will return aarch64 as existing
third party software expects this.

Differential Revision:	https://reviews.freebsd.org/D2005
Relnotes:	Yes
Sponsored by:	The FreeBSD Foundation
2015-03-19 13:53:47 +00:00
Baptiste Daroussin
8cf263c843 run pw_mkdb and cap_mkdb at post-installation for cross installation the version
from the host would be run but the generated files will be on the target
respecing the endianness of the target
2015-03-18 16:06:04 +00:00
Baptiste Daroussin
9269ba461b Always cleanup remnant plist in destdir if any 2015-03-18 11:26:10 +00:00
Baptiste Daroussin
ee1376bc63 Fix printing the package name when creating 2015-03-18 11:08:47 +00:00
Baptiste Daroussin
d3cd95cf88 Now that pkg create can directly accept both a file and a plist in command line,
directly use it
2015-03-18 09:39:31 +00:00
Baptiste Daroussin
9d73215448 Detect ABI based on what would be packaged 2015-03-18 05:25:38 +00:00
Ed Maste
a8bed44af5 When cross-building with an external toolchain we still need a target strip
It is used by at least crunchide(1).
2015-03-17 19:35:50 +00:00
Baptiste Daroussin
59fa1525e0 Merge from head 2015-03-17 19:10:51 +00:00
Bryan Drewery
215d02b78c Add LIB_CXX so that C++ libraries will use CXX to link.
This fixes C++ libraries not implicitly linking in libc++.  This is
generally not an issue because the final linking with the compiled binary
will involve CXX via PROG_CXX or other means.  It is however
inconsistent with libraries implicitly linking in libc and problematic
for trying to build libraries with '-z defs' to ensure all direct
dependencies are linked in.

libatf-c++ is currently the only consumer of this new feature.

Differential Revision:	https://reviews.freebsd.org/D2039
Reviewed by:	imp
Discussed with:	bapt
MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-03-17 15:16:36 +00:00
Baptiste Daroussin
5d3fd44bbf Directly use bmake's ':gmtime' 2015-03-15 14:41:26 +00:00
Baptiste Daroussin
42450b4928 Create a "packages" target takes care of all the magic 2015-03-15 14:05:55 +00:00
Baptiste Daroussin
7426d57242 Merge from head 2015-03-15 10:58:47 +00:00