This should be noticeably faster due to fewer processes being forked and
also handles other flags such as -S or writing to METALOG.
Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D26039
The most awkward bit in this patch is the bootstrapping of m4:
We can't simply use the host version of m4 since that is not compatible
with the flags passed by lex (at least on macOS, possibly also on Linux).
Therefore we need to bootstrap m4, but lex needs m4 to build and m4 also
depends on lex (which needs m4 to generate any files). To work around this
cyclic dependency we can build a bootstrap version of m4 (with pre-generated
files) then use that to build the real m4.
This patch also changes the xz/unxz/dd tools to always use the host version
since the version in the source tree cannot easily be bootstrapped on macOS
or Linux.
Reviewed By: brooks, imp (earlier version)
Differential Revision: https://reviews.freebsd.org/D25992
The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.
I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.
Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.
Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.
Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872
Although I can't reproduce it, others are seeing different lex/yacc
programs always regenerated after my change to copy rather than
symlink the files. The reported fix is to add '-p' to the copies.
Since it doesn't hurt, go head and add it, though the reasons for
this mattering remain at best obscure and poorly articulated.
Without this change the buildworld/buildkernel epilogue looks like this:
>>> World built in 249 seconds, sysctl: cannot stat /proc/sys/hw/ncpu: No such file or directory
ncpu: , make -j72.
Reviewed By: emaste, bdrewery
Differential Revision: https://reviews.freebsd.org/D26056
This ensure that running make install inside buildenv correctly includes
the METALOG flags when building with -DNO_ROOT.
Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D26038
upstream so Version.inc now only defines LLD_VERSION_STRING.
This breaks the WANT_LINKER_VERSION magic and might lead to us building
more than needed (e.g., for croos-tools).
Change the awk script to parse LLD_VERSION_STRING instead of LLD_VERSION,
which not only unbreaks the current situation but should also be backwards
compatible as dim points out.
PR: 248818
Reviewed by: emaste, dim (seems right and the way to go)
MFC after: 4 weeks
X-MFC before: 364284
Some of the scripts used for libdtrace invoke nawk instead of awk
(for example cddl/contrib/opensolaris/lib/libdtrace/common/mknames.sh).
When bootstrapping all tools, we get the nawk -> awk link while building
usr.bin/awk, but when linking/copying the dependencies from the host we
were only adding awk but not nawk.
This was silently generating invalid files when building libdtrace with
BUILD_WITH_STRICT_TMPPATH=1 since those scripts invoke nawk instead of
awk. In addition to adding the missing link this commit also adds
set -e to those scripts to catch errors like this in the future.
Reviewed By: markj, emaste
Differential Revision: https://reviews.freebsd.org/D26025
Since the make variable STRIP is already used for other purposes, this
uses STRIPBIN (which is also used for the same purpose by install(1).
This allows using LLVM objcopy to strip binaries instead of the in-tree
elftoolchain objcopy. We make use of this in CheriBSD since passing
binaries generated by our toolchain to elftoolchain strip sometimes results
in assertion failures.
This allows working around https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248516
by specifying STRIPBIN=/path/to/llvm-strip
Obtained from: CheriBSD
Reviewed By: emaste, brooks
Differential Revision: https://reviews.freebsd.org/D25988
make copies instead.
There's too many times that we can't run the new binaries with old
libraries. Making the links when things are known to be 'safe' is a
nice optimization, but a copy of all the binaries is only 30MB, so
saving the copies at the cost of increased support when new symbols
are added and used as part of the bootstrap seems to be unwise.
There may be additional optimizations possible here, especially for
!FreeBSD hosts. However, that's beyond the scope of the problem I'm
trying to fix with make failing mid-way through an installworld across
change r363679. This optimization there caused us to run a new binary
with an old library once a new make was installed due to the symbolic
link. One could just copy make, but then other binaries fail as well,
so rather than play whack-a-mole, I opted to take us back to the old
way. Before r340157 or so we did copies (thogh of a lot fewer
artifacts), and we didn't have issues like this.
Reviewed by: arichards@
Differential Revision: https://reviews.freebsd.org/D25967
The output dir is set to ${REPODIR}/${PKG_ABI}/${PKG_VERSION} now.
Add the possibility to specify the last componant and set it by default
to ${PKG_VERSION} as before.
This is useful for tests and also for building packages with the same
PKG_VERSION provided to check differences.
libarchive uses the SOURCE_DATE_EPOCH env variable to set the date of file
in an archive, this is useful for reproducibility.
Add a variable name PKG_TIMESTAMP that take a epoch time and set SOURCE_DATE_EPOCH
to this. By default it is the current time so no changes here.
In lib/Makefile, we document the dependency with SUBDIR_DEPEND
For buildworld orchestration, just prebuild libregex if GOOGLETEST is
enabled. googletest will get built in a later pass.
clang-format is enabled conditional on either WITH_CLANG_EXTRAS or
WITH_CLANG_FORMAT. Some sources in libclang are build conditional on
either rule, and obviously the clang-format binary itself depends on the
rule.
clang-format could still use a manual page.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D25427
As of r362452, liblzma depends on libmd but the buildworld build order
hadn't been amended to document the new dependency.
Reported by: jenkins via freqlabs
X-MFC-With: r362452
libucl comes with a Lua library binding. Build it into flua.
This lets us parse/generate config files in the various formats supported by
libucl with flua. For example, the following script will detect the format of
an object written to stdin as one of UCL config, JSON, or YAML and write it to
stdout as pretty-printed JSON:
local ucl = require('ucl')
local parser = ucl.parser()
parser:parse_string(io.read('*a'))
local obj = parser:get_object()
print(ucl.to_format(obj, 'json'))
Reviewed by: kevans, pstef
Approved by: mmacy (mentor)
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D25009
As of r361857 all BINUTILS options are disabled by default - ports
have been changed to depend on binutils if they require GNU as, and
all base system assembly files have been switched to use Clang's
integrated assembler.
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
-development is long and awkward, and is also inconsistent with prior art
from the Linux world, which uses -dev (Debian) or -devel (Red Hat). Follow
the Debian convention, and similarly for debug info packages.
Also remove redundant pkgbase development tag from includes. We already tag
include files with package=runtime,dev; there is no need to separately tag
them as dev.
Discussed with: bapt
Reviewed by: manu
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D24139
This was solving the correct-ish problem in the wrong place. Noted by
brooks; while he didn't request a prompt revert, doing so now will
facilitate proper testing for the revised version of this.
if foo; then blah else blah; fi has a drastically different meaning than
if foo; then blah; else blah; fi. Fix it.
Reported by: 0mp
X-MFC-With: r360833
This can be run as root or normal user with no problem; if they hadn't
twisted the WITHOUT_CAROOT knob, we'll attempt to use the host certctl to
rehash the DESTDIR. This would allow one to build systems WITHOUT_OPENSSL +
WITH_CAROOT with a populated /etc/ssl that they can then use with an
appropriate *ssl from somewhere else.
Cross-builds are fine because this will always use the host certctl, or just
nag if it's missing and it wasn't a WITHOUT_CAROOT build.
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D24641
The latter needs the former, but with a multi-job build on a fast
machine, the race is sometimes lost. This leads to "ld: error: unable to
find library -lsbuf", when linking libgeom.so.
Submitted by: kevans
MFC after: 3 days
r359461 introduced this nifty script to centralize these things, so add
shm_open.c there to remove a total of one (1) bad example from
Makefile.inc1.
Looked over by: emaste
instead of sprinkling them out over many disjoint files. This is a follow-up
to achieve the same goal in an incomplete rev.348521.
Approved by: imp
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D20520
By using -nobuiltininc and adding the clang builtin headers resource dir
to the end of the compiler header search path, we can still find headers
such as immintrin.h but find the FreeBSD version of stddef.h/stdarg.h/..
first.
This is a workaround until we are able to settle on and complete a plan
to harmonize guard macros with LLVM. We've mostly worked out this on
FreeBSD systems by removing select headers from the installed set of
devel/llvm*, but that isn't a good solution for cross build.
Submitted by: arichardson
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D17002
It's rather awkward to debug issues with the dependency cleanup hacks
when implemented via make. Add a cleanup shell script and move the
libomp hack there as an initial example.
Reviewed by: brooks
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D24228
After the base kyua import the testsuite can utilize the in-base kyua
just fine.
Submitted by: Dries Michiels
Differential Revision: https://reviews.freebsd.org/D24230
We need to build kyua libraries for kyua. Because we set MK_TESTS=no,
we can't not set MK_TESTS_SUPPORT=${MK_TESTS_SUPPORT} because the latter
defaults MK_TESTS_SUPPORT to no.
This fixes WITHOUT_TESTS + WITH_TESTS_SUPPORT builds.
Reported by: Ruslan Garipov
Reviewed by: emaste
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24206
Those libs aren't needed anymore so stop building them for the bootstrap tools
or the compat libs.
Reviewed by: bapt, emaste
X-Differential Revision: D24171
For the final step of the native-xtools target, "everything" is built
with TARGET and TARGET_ARCH set to the architecture we wish to
cross-build for. However, CROSSENV overwrites the values of MACHINE and
MACHINE_ARCH, setting them to be identical to TARGET and TARGET_ARCH.
For native-xtools this is undesirable since we are building binaries to
run on the host architecture, that can cross compile for the target
architecture.
When building native-xtools for RISC-V, this issue manifests as an invalid
argument for "-march". The compiler is invoked with the target triple of the
host architecture, but the CFLAGS inherited from bsd.cpu.mk are that of the
target architecture.
Reviewed by: imp, bdrewery
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D23838
We handle a number of stale dependency issues on an ad-hoc basis, in
order to support ongoing NO_CLEAN builds. These hacks do not need to
be maintained indefinitely; now remove those for issues that are more
than a year old.
Sponsored by: The FreeBSD Foundation
r359083 introduced a workaround for stale libomp dependencies during a
regular (no -DNO_CLEAN) buildworld. r359088 addressed the reason the
clean step missed libomp, so revert the workaround.
Sponsored by: The FreeBSD Foundation
Apparently make ${CLEANDIR} is leaving stale entries in .depend files;
for now invoke the hacky cleanup in both the -DNO_CLEAN and normal
(no -DNO_CLEAN) cases.
In collaboration with: dim
Sponsored by: The FreeBSD Foundation
The new liblua will be used in a forthcoming import of kyua.
Reviewed by: kevans
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24090
We have ad-hoc stale dependency handling in Makefile.inc1 to handle the
cases where file extensions change, but it appears that some cases are
not functional. Add a note when about to clean stale deps to help
when investigating failure reports.
Sponsored by: The FreeBSD Foundation
After r325072 stale lib32 dependencies were not remooved. A more
holistic approach is needed to address this but for the immediate issue
(-DNO_CLEAN builds across r358851) just readd the explicit lib32 path.
Reported by: dim
Sponsored by: The FreeBSD Foundation
