This is because the .meta files generated from filemon already contain a
list of all files read to generate the object.
X-MFC-With: r290433
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
This is especially noticeable in the kernel obj directory since it
includes so many files.
X-MFC-With: r290433
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
This speeds up buildworld by 16% on my system and buildkernel by 35%.
Rather than calling mkdep(1), which is just a wrapper around 'cc -E',
use the modern -MD -MT -MF flags to gather and generate dependencies during
compilation. This flag was introduced in GCC "a long time ago", in GCC 3.0,
and is also supported by Clang. (It appears that ICC also supports this but I
do not have access to test it). This avoids running the preprocessor *twice*
for every build, in both 'make depend' and 'make all'. This is especially
noticeable when using ccache since it does not cache preprocessor results from
mkdep(1) / 'cc -E', but still speeds up compilation with the -MD flags.
For 'make depend' a tree-walk is still done to ensure that all DPSRCS
are generated when expected, and that beforedepend/afterdepend and
_EXTRADEPEND are all still respected. In time this may change but for now
I've been conservative. The time for a tree-walk with -j combined with
SUBDIR_PARALLEL is not significant. For example, it takes about 9 seconds
with -j15 to walk all of src/ for 'make depend' now on my system.
A .depend file is still generated with the various rules that apply to
the final target, or custom rules. Otherwise there are now
per-built-object-file .depend files, such as .depend.filename.o. These
are included directly by make rather than populating .depend with a loop
and .depend lines, which only added overhead to the now almost-NOP 'make
depend' phase.
Before this I experimented with having mkdep(1) called in parallel per-file.
While this improved the kernel and lib/libc 'make depend' phase, it resulted
in slower build times overall.
The -M flags are removed from CFLAGS when linking since they have no effect.
Enabling this by default, for src or out-of-src, can be done once more testing
has been done, such as a ports exp-run, and with more compilers.
The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_FAST_DEPEND=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16
buildworld:
x buildworld-before
+ buildworld-fastdep
+-------------------------------------------------------------------------------+
|+ |
|+ |
|+ xx x|
| |_MA___||
|A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)
buildkernel:
x buildkernel-before
+ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|+ x |
|++ xx|
| A||
|A| |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)
Sponsored by: EMC / Isilon Storage Division
MFC after: 3 weeks
Relnotes: yes
an error.
Most of these do a 'mkdir -p' or 'install -d' before installing, but add
the trailing / here for consistency with the userland install.
MFC after: 2 weeks
X-MFC-With: r289391
Sponsored by: EMC / Isilon Storage Division
- Move the required kernel compiler flags from Makefile.arm64 to kern.mk.
- Build arm64 modules as PIC; non-PIC relocations in .o for shared object
output cannot be handled.
- Do not try to install aarch64 symlink.
- A hack for arm64 to avoid ld -r stage. See the comment for the explanation.
Some functionality is lost, like ctf handling, but hopefully will be
restored after newer linker is available.
Reviewed by: andrew, emaste
Tested by: andrew (on real hardware)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3796
This avoids needing a large boot partition / file system in order to
accommodate multiple kernels, and provides consistency with userland
debug. This also simplifies the process of moving kernel debug files
to a separate package and installing them on demand.
In addition, change kernel debug file extension to .debug, to match
userland debug files.
When using the supported kernel installation method the
/usr/lib/debug/boot/kernel directory will be renamed (to kernel.old)
as is done with /boot/kernel.
Developers wishing to maintain the historical behavior of installing
debug files in /boot/kernel/ can set KERN_DEBUGDIR="" in src.conf(5).
Reviewed by: bdrewery, brooks, imp, markj
Relnotes: yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D1006
root disk. The embedded image is linked into the kernel in the .mfs
section.
Add rules and variables to kern.pre.mk and kern.post.mk that handle the
linking of the image. First objcopy is used to generate an object file.
Then, the object file is linked into the kernel.
Submitted by: Steve Kiernan <stevek@juniper.net>
Reviewed by: brooks@
Obtained from: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D2903
This will require for AArch64 as we dont have modules yet.
Sponsored by: HEIF5
Sponsored by: ARM Ltd.
Differential Revision: https://reviews.freebsd.org/D1997
Make the process of embedding MDROOT images less perilous by
makeing the target that links kernel and embedding the image
depend on the image. This means, if the image doesn't exist you
find out before you try to boot from it and that if you change
the image you don't have to touch some random source file to
cause a rebuild.
Don't hide that we're embedding the image.
the aggressive pattern matching of the :C modifier. I tested build and
install in 2 phases, however with different solutions, resulting in the
breakage. Mea culpa.
The solution is to break out the all: target. This causes a few lines of
code duplication, but now the all: target works as it should, and the
other targets continue to work as they did before.
While I'm here, add a ===> header line to the start of each port build
to make it easier to find/more clear in the logs.
important for those that use -DNO_CLEAN routinely, since it will prevent
installing stale stuff, and even more important when the port is upgraded
to a newer version. When the user doesn't use -DNO_CLEAN, this will create
an infinitesimal amount of extra work, but won't hurt anything.
This is necessary because the ports tree has flags that prevent the ususal
'update the build if newer source files exist' logic from doing what it
would do in the base.
LOCALBASE/bin and sbin to PATH, allowing dependencies to be found;
adding SRC_BASE and OSVERSION to match the new kernel, and putting
the related builds under MAKEOBJDIRPREFIX so that they only need
to be built once per kernel.
In addition to the PR this includes ideas/contributions from crees
and matthew.
PR: ports/161452
Submitted by: Garrett Cooper <yanegomi@gmail.com>
The 'make depend' rules have to use custom -I paths for the special compat
includes for the opensolaris/zfs headers.
This option will pull in the couple of files that are shared with dtrace,
but they appear to correctly use the MODULE_VERSION/MODULE_DEPEND rules
so loader should do the right thing, as should kldload.
Reviewed by: pjd (glanced at)
defined based on WITH/WITHOUT_CTF settings, default is WITHOUT_CTF,
NO_CTF overrides WITH_CTF (used by Makefile.inc1)
- CTFCONVERT_CMD/NORMAL_CTFCONVERT are now defined to empty string
if make(1) can handle empty commands
- CTFCONVERT_CMD=... is a hack (should be defined to empty string instead):
make(1) should be taught to ignore empty commands silently in compat mode
(as it does in !compat mode, GNU make also silently ignores empty commands)
and to skip printing empty commands in !compat mode
- config(8) should generate ${NORMAL_CTFCONVERT} invocation without '@':
this will allow to simplify kern.pre.mk even more and lessen the number
of shell invocations during kernel build when CTF is turned off
- WITH_CTF can now be converted to usual MK_CTF=yes/no infrastructure
Pointy hat to: fjoe [1]
wihtout updating world (good transition aide for -current, but also
allows kernels to be built on -stable the old way too). This likely
should go away around FreeBSD 10.0 or so.
WITH{OUT,}_KERNEL_SYMBOLS (defaulting to WITH). In the fullness of
time, likely around 2020, INSTALL_NODEBUG will be removed. For now,
don't print a warning when using INSTALL_NODEBUG, but that will be
coming soon.
to amd64, i386, and pc98. The headers are installed to /usr/include/x86
during an installworld, and an 'x86' symlink is created for kernel builds
similar to 'machine' so that the headers can be included as <x86/foo.h>.
Reviewed by: imp
are some problems with static executables), make.conf (would also
affect ports which do not use GNU make and do not override the
compile targets) or in the kernel config (via "makeoptions
WITH_CTF=yes").
Additional (related) changes:
- propagate WITH_CTF to module builds
- do not add -g to the linker flags, it's a noop there anyway
(at least according to the man page of ld)
- do not add -g to CFLAGS unconditionally
we need to have a look if it is really needed (IMO not) or if there
is a way to add it only when WITH_CTF is used
Note: ctfconvert / ctfmerge lines will not appear in the build output,
to protect the innocent (those which do not build with WITH_CTF would
see the shell-test and may think WITH_CTF is used).
Reviewed by: imp, jhb, scottl (earlier version)
Discussed on: arch@
to module builds. This avoids having to have the module builds walk up
the tree to find the kernel sources. It also allows a kernel + module
build to succeed when a new level of module subdirectories is added without
requiring that the /usr/share/mk/bsd.kmod.mk file on the machine be patched.
MFC after: 1 week
installing the kernel allows one, like with modules, to override
the default user/group and install as non-su to a temporary
directory to test, create images or seed a tftp dir.
Reviewed by: Andrzej Tobola <ato@iem.pw.edu.pl>
MFC after: 4 weeks
This allows to fix a problem with ARM kernel.bin not having the MFS image
embedded: it is objcopied from the kernel.noheader temporary ELF file, which
was not subject to embedding the MFS image previously.
Reviewed by: imp
Approved by: cognet (mentor)
for those upgrading from 4.x -> 5.x. It is therefore no longer
necessary because it is installed on 5.x and 6.x and one has to
upgrade from 6.x -> current, or at the very least 5.x (although not
supported, it can be made to work).
only now) symbolic links in the kernel compile directory, rather
than relying on config(8) to do this. (The changes to config(8)
will be committed separately.) This is aimed towards making the
config(8) as lightweight as possible.
Idea by: bde (all bugs are mine)
gdb(1) command better, though I must admit it's confusing: these
files have not only [debugging] symbols, but much more than that.
Requested by: obrien
our kernel linker will only load PT_LOAD segments, apparently not.
Instead, produce .dbg objects from .debug objects, and install
them together with non-debug objects, as described in objcopy(1).
Original code by: obrien
Try to make everyone happy: David (to have debug kernels installed
by default), Warner (to be able to override that), and myself (for
actually making it all work and to be consistent).
Now, if kernel was configured for debugging (through DEBUG=-g in
the kernel config file or "config -g"), doing "make install" will
install debug versions of kernel and module objects with their
canonical names,
kernel.debug -> /boot/kernel/kernel
if_fxp.ko.debug -> /boot/kernel/if_fxp.ko
Installing a kernel not configured for debugging, or debug kernel
with INSTALL_NODEBUG variable defined, will install non-debug
kernel and module objects.
Also, restore the install.debug and reinstall.debug targets that
are part of the existing API (they cause some additional gdb(1)
scripts to be installed).
be installed. It should have been optional to install a non-debug
one, just like it was formerly optional to install a debug one. In
order to do that, most of 1.84 had to go.
Instead, make installing the debug kernel the default, but create a
new option INSTALL_NODEBUG for those people that have small /
partitions and good source control habits.
This preserves the behavior of 1.84 while allowing it to be overriden
for people (like me) that do not have the time to upgrade to get a
bigger / and also don't have time for stupid makefile tricks when
upgrading their older system, but still want a kernel.debug around if
things go south.
initializations but we did have lofty goals and big ideals.
Adjust to more contemporary circumstances and gain type checking.
Replace the entire vop_t frobbing thing with properly typed
structures. The only casualty is that we can not add a new
VOP_ method with a loadable module. History has not given
us reason to belive this would ever be feasible in the the
first place.
Eliminate in toto VOCALL(), vop_t, VNODEOP_SET() etc.
Give coda correct prototypes and function definitions for
all vop_()s.
Generate a bit more data from the vnode_if.src file: a
struct vop_vector and protype typedefs for all vop methods.
Add a new vop_bypass() and make vop_default be a pointer
to another struct vop_vector.
Remove a lot of vfs_init since vop_vector is ready to use
from the compiler.
Cast various vop_mumble() to void * with uppercase name,
for instance VOP_PANIC, VOP_NULL etc.
Implement VCALL() by making vdesc_offset the offsetof() the
relevant function pointer in vop_vector. This is disgusting
but since the code is generated by a script comparatively
safe. The alternative for nullfs etc. would be much worse.
Fix up all vnode method vectors to remove casts so they
become typesafe. (The bulk of this is generated by scripts)
Users should move to the new geom_vinum implementation instead.
The refcount logic which is being added to devices to enable safe module
unloading and the buf/vm work also in progress would require a major rework
of the (old)-vinum code to comply with the new semantics.
The actual source files will not be removed until I have coordinated with
the geomvinum people if they need any bits repo-copied etc.
This is necessary so source upgrades use the correct binary.
MFC after: 3 days
For the record: Problem spotted by Scott Long, who mentioned
that source upgrades from 4.7 to recent 5.x and 6.0 are broken.
Detailed analysis shows that 4.7 has a broken make(1) binary.
A breakage was fixed in RELENG_4 in make/main.c,v 1.35.2.7 by
imp@, though the commit log erroneously stated "MFC 1.68"
while in fact it should have been spelled as "MFC 1.67".
Now that the devs files are marked before-depend, we can remvoe them
from a few places they were explicitly mentioned (along with
BEFORE_DEPEND).
Noticed by: bde
delete it each time its run and have it regenerated each time by make.
I used a quick hackish script rather than putting it in the files file
and used the before-depend rule to avoid the depend/no-depend hacks.
uncovering some interesting problems. Be conservative and effecitvely
disable this by default. Interested parties may still define
KERNBUILDDIR by hand to achive the same effect.
I plan on referting this change after 5.2 is released, or sooner if
the issues with building releases are resolved and re@ approves.
Approved by: re@ (scottl, marcel)
kernel build. This makes it possible for me not to get pissed off that
random.ko crashes the system trying to rdtsc() when the i386/cpu.h
support code decides it's okay to call that op when neither I386_CPU or
I486_CPU is defined. I guess it also makes WITNESS/INVARIANTS defines
get picked up by the modules.
no matter where in the directory structure it may be. Use this and the "-k"
flag in the generated gdbinit files so that the "getsyms" function in gdb
requires no user intervention to run and will find every module if they're
in the kernel build's module directory. This is still quite useful for
cases where gdb knows that the path for some modules is /boot/kernel and
others are in the object directory for /usr/src/sys/$ARCH/compile/kernel.
Approved by: grog
Always use sys/conf/kern.mk when building kernel/modules.
<bsd.kern.mk> is only preserved for sys/boot/pc98/boot2
for now, but this will be fixed. If there are other
users of <bsd.kern.mk>, please let me know.
Reminded by: bde
is a compiler tool and needs to be compiled by the host compiler. I've
tested this in i386->sparc cross-build, 4.7->current upgrade, normal
buildkernel target, and normal /sys/i386/compile/GENERIC configurations.
Submitted by: ru
under way to move the remnants of the a.out toolchain to ports. As the
comment in src/Makefile said, this stuff is deprecated and one should not
expect this to remain beyond 4.0-REL. It has already lasted WAY beyond
that.
Notable exceptions:
gcc - I have not touched the a.out generation stuff there.
ldd/ldconfig - still have some code to interface with a.out rtld.
old as/ld/etc - I have not removed these yet, pending their move to ports.
some includes - necessary for ldd/ldconfig for now.
Tested on: i386 (extensively), alpha
the environment for the last command of the pipeline (xargs) instead
of too early in the broken version or using an extra env process for
each command spawned by xargs as in rev.1.12. Fixed a nearby English
error.
-finstrument-functions instead of -mprofiler-epilogue. The former
works essentially the same as the latter but has a higher overhead
(about 22 more bytes per function for passing unused args to the
profiling functions).
Removed all traces of the IDENT Makefile variable, which had been
reduced to just a place for holding profiling's contribution to CFLAGS
(the IDENT that gives the kernel identity was renamed to KERN_IDENT).
to build kernel and kernel modules so stop supporting them in
bsd.subdir.mk and reimplement them in kern.post.mk and kmod.mk
as special versions of the install and reinstall targets, and
only define them if DEBUG is also defined (when debug versions
are really built).
Prompted by: bde
"env name=value ... cmd ..." is just a pessimized way of doing
"name=value ... cmd ..." in real shells. Set the environment
(without using env(1)) before starting xargs so that env(1)
is not needed in "xargs env name=value ... cmd ..."
lint, so this is turned off by default. Setting WANT_LINT will turn
on generation of lint libraries for /usr/libdata/lint/*.ln.
Reviewd by: silence in -audit.
so that this is safe even if VARIABLE is longer than kern.argmax.
There is another instance of CFILES which might need the same treatment,
and might be noticed when doing a "make links".
The same has to be done in RELENG_4 (on some different file).
Noticed-by: picobsd cross-compiling LINT
Suggested-by: Alfred (bright@mu.org), des@freebsd.org
MFC-after: 3 days
simplifying the module linking process and eliminating the risks
associated with doubly-defined variables.
Cases where commons were legitimately used (detection of
compiled-in subsystems) have been converted to use sysinits, and
any new code should use this or an equivalent practice as a
matter of course.
Modules can override this behaviour by substituting -fno-common
out of ${CFLAGS} in cases where commons are necessary
(eg. third-party object modules). Commons will be resolved and
allocated space when the kld is linked as part of the module
build process, so they will not pose a risk to the kernel or
other modules.
Provide a mechanism for controlling the export of symbols from
the module namespace. The EXPORT_SYMS variable may be set in the
Makefile to NO (export no symbols), a list of symbols to export,
or the name of a file containing a newline-seperated list of
symbols to be exported. Non-exported symbols are converted to
local symbols. If EXPORT_SYMS is not set, all global symbols are
currently exported. This behaviour is expected to change (to
exporting no symbols) once modules have been converted.
Reviewed by: peter (in principle)
Obtained from: green (kmod_syms.awk)
kern.post.mk.
# this should allow us to move kern.post.mk to the last line of the makefiles,
# but I'll do that slowly as I verify that one can do that w/o breaking things.
Submitted by: naddy
Fix it by putting back the link of machine to sys/i386/include rather
than ../../include (aka sys/pc98/include). I had a stale machine link
on my first test.
Not sure what the "right" fix is, but this unbreaks things.
new files: kern.pre.mk, which contains most of the definitions, and
kern.post.mk, which contains most of the rules.
I've tested this on i386 and pc98. I have had feedback on the sparc64
port, but no reports from anybody on alpha, ia64 or powerpc. I
appologize in advance if I've broken you.
Reviewed by: jake, jhb, arch@
