As discussed on -current, -stable, -toolchain, and with jhb@ and imp@,
disable the obsolete in-tree GDB 6.1.1 by default. This was kept only
to provide kgdb for the crashinfo tool, but is long-obsolete, does not
support all architectures that FreeBSD does, and held back other work
(such as forcing the use of DWARF2 for kernel debug).
Crashinfo will use kgdb from the gdb package or devel/gdb port, and will
privde a message referencing those if no kgdb is found.
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
This has been years in the making, and we all knew it was bound to happen
some day. Switch to the BSDL grep implementation now that it's been a
little more thoroughly tested and theoretically supports all of the
extensions that gnugrep in base had with our libregex(3).
Folks shouldn't really notice much from this update; bsdgrep is slower than
gnugrep, but this is currently the price to pay for fewer bugs. Those
dissatisfied with the speed of grep and in need of a faster implementation
should check out what textproc/ripgrep and textproc/the_silver_searcher
can do for them.
I have some WIP to make bsdgrep faster, but do not consider it a blocker
when compared to the pros of switching now (aforementioned bugs, licensing).
PR: 228798 (exp-run)
PR: 128645, 156704, 166842, 166862, 180937, 193835, 201650
PR: 232565, 242308, 246000, 251081, 191086, 194397
Relnotes: yes, please
This was introduced and then disabled by default primarily to avoid dealing
with bugs in libgnuregex. rS363823 switched to using libregex for it, so
let's just rip the option out now so we can make sure we're getting tested
with libregex via bsdgrep.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D27476
We are seeing regular build failures due to libc.so being installed again and
another parallel make job tries to read the partially written libc.so at the
same time. When building with -j32 or higher this almost always happens on
the first clean build (subsequent incremental builds always work fine).
Using -S should "fix" the "section header table goes past the end of the
file: e_shoff = 0x..." errors that have started to plague our builds.
We originally thought this only affected CheriBSD, but I just got the same
error while building the latest upstream FreeBSD.
The real fix should be to not install libraries twice, but until then this
workaround is needed.
Original patch by jrtc27@, I only made some minor changes to the comment.
Obtained from: CheriBSD (49837edd3e)
Reviewed By: markj, bdrewery
Differential Revision: https://reviews.freebsd.org/D27102
Check for the variable SUBDIR. and error as it usually means someone
forgot to include src.opts.mk.
This guard from CheriBSD found the bugs in r367655 and r367728.
Reviewed by: bdrewery, arichardson
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D27211
WITH_INIT_ALL_ZERO and WITH_INIT_ALL_PATTERN are mutually exclusive.
The .error when they were both set broke makeman so demote it to a
warning (and presumably the compiler will fail on an error later on).
We could improve this to make one take precedence but this is sufficient
for now.
MFC with: r367577
Sponsored by: The FreeBSD Foundation
There are two options:
- WITH_INIT_ALL_ZERO: Zero all variables on the stack.
- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns.
The exact pattern are a compiler implementation detail and vary by type.
They are somewhat documented in the LLVM commit message:
https://reviews.llvm.org/rL349442
I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather
than naming them after the LLVM specific compiler flags.
In a range of consumer products, options like these are used in
both debug and production builds with debugs builds using patterns
(intended to provoke crashes on use of uninitialized values) and
production using zeros (deemed more likely to lead to harmless
misbehavior or NULL-pointer dereferences).
Reviewed by: emaste
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D27131
LLVM's demangler supports more modern C++ constructs such as lambdas and
unnamed types, and is actively maintained. The command line tool is
usable as a drop-in replacement for GNU c++filt, or elftoolchain's
cxxfilt. The latter is still available by using WITHOUT_LLVM_CXXFILT, if
needed.
PR: 250702
MFC after: 2 weeks
Since elftoolchain's cxxfilt is rather far behind on features, and we
ran into several bugs, add an option to use llvm-cxxfilt as an drop-in
replacement.
It supports the same options as elftoolchain cxxfilt, though it doesn't
have support for old ARM (C++ Annotated Reference Manual, not the CPU)
and GNU v2 manglings. But these are irrelevant in 2020.
Note: as we already compile the required libraries as part of libllvm,
this will not add any significant build time either.
PR: 250702
Reviewed by: emaste, yuri
Differential Revision: https://reviews.freebsd.org/D27071
MFC after: 2 weeks
Until clang 11 that was equivalent to -O2, but clang changed it to -O1 so
generated MIPS code will now be unnecessarily slow. It also removes a weird
special case from sys.mk.
This is similar to the D26471 change for debug kernels and should not change
anything since everything was previously building MIPS code at -O2 until the
clang 11 update.
Reviewed By: trasz
Differential Revision: https://reviews.freebsd.org/D26749
It appears this was changed from ln to use install in rS245752. I noticed
this because my buildenv was setting INSTALL=install -U -M //METALOG
and then these links fail to be created with the following error:
install: open //METALOG: Permission denied
Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D26618
Add a wrapping script to use ATF to run tests written with Googletest
one by one. This helps locating and tracking the failing case in CI easier.
This is a temporarily solution while Googletest support in Kyua is developing.
We will revert this once Kyua+Googletest integration is ready.
Reviewed by: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D25896
VirtFS allows sharing an arbitrary directory tree between bhyve virtual
machine and the host. Current implementation has a fairly complete support
for 9P2000.L protocol, except for the extended attribute support. It has
been verified to work with the qemu-kvm hypervisor.
Reviewed by: rgrimes, emaste, jhb, trasz
Approved by: trasz (mentor)
MFC after: 1 month
Relnotes: yes
Sponsored by: Conclusive Engineering (development), vStack.com (funding)
Differential Revision: https://reviews.freebsd.org/D10335
Clang 12 warns about passing a path to -fuse-ld and -Werror makes that
an error preventing building world without this change.
Reviewed by: arichardson, emaste
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D26591
This is the initial set up for PowerPC64LE.
The current plan is for this arch to remain experimental for FreeBSD 13.
This started as a weekend learning project for me and kinda snowballed from
there.
(More to follow momentarily.)
Reviewed by: imp (earlier version), emaste
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D26399
bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.
By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.
By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.
bootonce dataset name is recorded in boot pool labels, bootenv area.
in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.
bootonce and nextboot features are usable in both UEFI and BIOS boot.
To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).
At this time, only lua loader is updated.
Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25512
As we do for shared library binaries, pass -S to install(1) when
installing symlinks. Doing so helps avoid transient failures when
libraries are being reinstalled, which seems to be the root cause of
spurious libgcc_s.so link failures during CI builds.
PR: 233769
Reviewed by: emaste
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26453
The current default is provided in various Makefile.inc in some top-level
directories and covers a good portion of the tree, but doesn't cover parts
of the build a little deeper (e.g. libcasper).
Provide a default in src.sys.mk and set WARNS to it in bsd.sys.mk if that
variable is defined. This lets us relatively cleanly provide a default WARNS
no matter where you're building in the src tree without breaking things
outside of the tree.
Crunchgen has been updated as a bootstrap tool to work on this change
because it needs r365605 at a minimum to succeed. The cleanup necessary to
successfully walk over this change on WITHOUT_CLEAN builds has been added.
There is a supplemental project to this to list all of the warnings that are
encountered when the environment has WARNS=6 NO_WERROR=yes:
https://warns.kevans.dev -- this project will hopefully eventually go away
in favor of CI doing a much better job than it.
Reviewed by: emaste, brooks, ngie (all earlier version)
Reviewed by: emaste, arichardson (depend-cleanup.sh change)
Differential Revision: https://reviews.freebsd.org/D26455
Use of ranlib or lorder is no longer necessary with current linkers
(probably anything newer than ~1990) and ar's ability to create an object
index and symbol table in the archive.
Currently the build system uses lorder+tsort to sort the .o files in
dependency order so that a single-pass linker can use them. However,
we can use the -s flag to ar to add an index to the .a file which makes
lorder unnecessary.
Running ar -s is equivalent to running ranlib afterwards, so we can also
skip the ranlib invocation.
Similarly, we don't have to pass the .o files for shared libraries in
dependency order since both ld.bfd and ld.lld will correctly resolve
references between the .o files.
This removes many fork()+execve calls for each library so should speed up
builds a bit. Additionally lorder.sh uses a regular expression that is not
supported by the macOS libc or glibc and results in many warnings when
cross-building (see D25989).
There is one functional change: lorder.sh removed duplicated .o files
from the linker command line which now no longer happens. I fixed the duplicates
in the base system in r364649. I also checked the ports tree for uses of
bsd.lib.mk and found one duplicate source file which I fixed in r548168.
Most ports use CMake/autotools rather than bsd.lib.mk but if this breaks any
ports that I missed in my search please let me know.
Avoiding the shell script actually speeds up the linking step noticeably: I
measured how long it takes to rebuild the .a and .so files for lib/libc using a
basic benchmark: `rm $LIBC_OBJDIR/*.so* $LIBC_OBJDIR/*.a* && /usr/bin/time make -DWITHOUT_TESTS -s > /dev/null`
Without this change ~4.5 seconds and afterwards ~3.1 seconds.
Looking at truss -cf output we can see that the number fork() system
calls goes down from 27 to 12 (and the speedup while tracing is more
noticeable: 81 seconds -> 65 seconds).
See also https://www.gnu.org/software/coreutils/manual/html_node/tsort-background.html
for some more background:
This whole procedure has been obsolete since about 1980, because Unix
archives now contain a symbol table (traditionally built by ranlib, now
generally built by ar itself), and the Unix linker uses the symbol table
to effectively make multiple passes over an archive file.
Or alternatively https://www.unix.com/man-page/osf1/1/lorder/:
The lorder command is essentially obsolete. Use the following command in
its place: % ar -ts file.a
Reviewed By: emaste, imp, dim
Differential Revision: https://reviews.freebsd.org/D26044
In D12421, the ability to compile stand/ in little-endian was added, with the
intention to extend loader.kboot to run in Petitboot.
However, no further work was done, as the kernel then gained self-execution
capabilities as Petitboot was taught to load FreeBSD kernels directly.
The FreeBSD installer on powerpc64 (on POWER8 and POWER9) uses
/boot/etc/kboot.conf instead of loader.
As this option does nothing but cause stand/ to be miscompiled and actively
causes confusion, remove it.
(I have a functioning petitboot loader in my local tree, however, it turned
out to be quite inconvient to use due to the current petitboot plugin design
so I put it on hold.)
Reviewed by: emaste, imp, jhibbits
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D26430
Use MACHINE_CPUARCH with arm64 (aarch64) when we build code that could run
on any 64-bit Arm instruction set. This will simplify checks in downstream
consumers targeting prototype instruction sets.
The only place we check for MACHINE_ARCH == aarch64 is when building the
device tree blobs. As these are targeting current generation ISAs.
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D26370
This allows use of the standard src.conf configuration for controlling
whether the tree is cleaned before build or not. The default is still
to clean.
Setting either NOCLEAN or NO_CLEAN will mention the new src.conf option.
NOCLEAN remains a .warning, while for now NO_CLEAN is .info.
Reviewed by: bdrewery (earlier version)
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D22762
For historical reasons, defining MALLOC_PRODUCTION in /etc/make.conf has
been used to turn off potentially expensive debug checks and statistics
gathering in the implementation of malloc(3).
It seems more consistent to turn this into a regular src.conf(5) option,
e.g. WITH_MALLOC_PRODUCTION / WITHOUT_MALLOC_PRODUCTION. This can then
be toggled similar to any other source build option, and turned on or
off by default for e.g. stable branches.
Reviewed by: imp, #manpages
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D26337
ld.bfd in particular requires -lm to come after libifconfig on the
command line when linking rescue.
Reviewed by: freqlabs, adrian
Differential Revision: https://reviews.freebsd.org/D26258
Allow architecture names to be passed in to the build system via CPUTYPE.
This allows the user to use values such as armv8.1-a or armv8-a+crc as
the CPUTYPE.
Sponsored by: Innovate UK
This changeset introduces the new libnetmap library for writing
netmap applications.
Before libnetmap, applications could either use the kernel API
directly (e.g. NIOCREGIF/NIOCCTRL) or the simple header-only-library
netmap_user.h (e.g. nm_open(), nm_close(), nm_mmap() etc.)
The new library offers more functionalities than netmap_user.h:
- Support for complex netmap options, such as external memory
allocators or per-buffer offsets. This opens the way to future
extensions.
- More flexibility in the netmap port bind options, such as
non-numeric names for pipes, or the ability to specify the netmap
allocator that must be used for a given port.
- Automatic tracking of the netmap memory regions in use across the
open ports.
At the moment there is no man page, but the libnetmap.h header file
has in-depth documentation.
Reviewed by: hrs
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D26171
I noticed that when we build libraries for a different ABI (in CheriBSD) we
were calling ${XCC}/${LD} --version for every directory. It turns out that
this was caused by bsd.compat.mk explicitly setting (X_)COMPILER variables
for that build stage and this stops the _can_export logic from working.
To fix this, we change the check to only set _can_export=no if the variable
is set and it is set to a different value than the cached value.
This noticeably speeds up the tree walk while building compat libraries.
During an upstream amd64 buildworld this also removes 8 --version calls.
Obtained from: CheriBSD
Reviewed By: brooks, emaste
Differential Revision: https://reviews.freebsd.org/D25986
When using relative paths for the linker we have to transform the name
since clang does not like -fuse-ld=ld.lld and instead requires -fuse-ld=lld
(the same also applies for ld.bfd).
This is needed so that setting LD/XLD is not ignored when linking with $CC
instead of directly using $LD. Currently only clang accepts an absolute
path for -fuse-ld= (Clang 12+ will add a new --ld-path flag), so we now
warn when building with GCC and $LD != "ld" since that might result in the
wrong linker being used.
We have been setting XLD=/path/to/cheri/ld.lld in CheriBSD for a long time and
used a similar version of this patch to avoid linking with /usr/bin/ld.
This change is also required when building FreeBSD on an Ubuntu with Clang:
In that case we set XCC=/usr/lib/llvm-10/bin/clang and since
/usr/lib/llvm-10/bin/ does not contain a "ld" binary the build fails with
`clang: error: unable to execute command: Executable "ld" doesn't exist!`
unless we pass -fuse-ld=/usr/lib/llvm-10/bin/ld.lld.
This change passes -fuse-ld instead of copying ${XLD} to WOLRDTMP/bin/ld
since then we would have to ensure that this file does not exist while
building the bootstrap tools. The cross-linker might not be compatible with
the host linker (e.g. when building on macos: host-linker= Mach-O /usr/bin/ld,
cross-linker=LLVM ld.lld).
Reviewed By: brooks, emaste
Differential Revision: https://reviews.freebsd.org/D26055
The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.
I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.
Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.
Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.
Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872
This is only needed when linking and fixes various "unused command
line argument" warnings during the lib32 build.
Reviewed by: bdrewery
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D26068
This is not strictly required for crossbuilding but having lots of warnings
from bsd.linker.mk in the output was making it hard to see the actual
warning messages.
Reviewed By: imp
Differential Revision: https://reviews.freebsd.org/D14318
These tools require a bootstrap llvm-tblgen/clang-tblgen and that cannot
be built with the current make infrastructure: the config header is not
correct for Linux/macOS and we don't include the CMakeLists.txt in contrib
so we can't generate one that would be correct.
Reviewed By: emaste, imp, dim
Differential Revision: https://reviews.freebsd.org/D14245
Since the make variable STRIP is already used for other purposes, this
uses STRIPBIN (which is also used for the same purpose by install(1).
This allows using LLVM objcopy to strip binaries instead of the in-tree
elftoolchain objcopy. We make use of this in CheriBSD since passing
binaries generated by our toolchain to elftoolchain strip sometimes results
in assertion failures.
This allows working around https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248516
by specifying STRIPBIN=/path/to/llvm-strip
Obtained from: CheriBSD
Reviewed By: emaste, brooks
Differential Revision: https://reviews.freebsd.org/D25988
gtest tests want to use \w ([[:alnum:]]) at the very least, which was
causing them to fail after r363679.
Start linking against libregex so that this shorthand is implemented.
PR: 248452
This implementation doesn't have any major deviations from the other EFI
ports. I've copied the boilerplate from arm and arm64.
I've tested this with the following boot flows:
OpenSBI (M-mode) -> u-boot (S-mode) -> loader.efi -> FreeBSD
OpenSBI (M-mode) -> u-boot (S-mode) -> boot1.efi -> loader.efi -> FreeBSD
Due to the way that u-boot handles secondary CPUs, OpenSBI >= v0.7 is required,
as the HSM extension is needed to bring them up explicitly. Because of this,
using BBL as the SBI implementation will not be possible. Additionally, there
are a few recent u-boot changes that are required as well, all of which will be
present in the upcoming v2020.07 release.
Looks good: emaste
Differential Revision: https://reviews.freebsd.org/D25135
These implementations of the bc and dc programs offer a number of advantages
compared to the current implementations in the FreeBSD base system:
- They do not depend on external large number functions (i.e. no dependency
on OpenSSL or any other large number library)
- They implements all features found in GNU bc/dc (with the exception of
the forking of sub-processes, which the author of this version considers
as a security issue).
- They are significantly faster than the current code in base (more than
2 orders of magnitude in some of my tests, e.g. for 12345^100000).
- They should be fully compatible with all features and the behavior of the
current implementations in FreeBSD (not formally verified).
- They support POSIX message catalogs and come with localized messages in
Chinese, Dutch, English, French, German, Japanese, Polish, Portugueze,
and Russian.
- They offer very detailed man-pages that provide far more information than
the current ones.
The upstream sources contain a large number of tests, which are not
imported with this commit. They could be integrated into our test
framework at a latter time.
Installation of this version is controlled by the option "MK_GH_BC=yes".
This option will be set to yes by default in 13-CURRENT, but will be off
by default in 12-STABLE.
Approved by: imp
Obtained from: https://git.yzena.com/gavin/bc
MFC after: 4 weeks
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D19982
