This reworks my last commit in r301285 to more closely match what was in
r241298 (but reverted in r294878).
This is addressing "missing .meta file" rebuilds but also ensuring that
files are always generated when needed in each case.
Note that this is not a complete rework of the problem areas identified
in r301285 as most are "good enough" right now as the new pattern
is too verbose. It's only worth making this current change where headers
may be generated in the INCS list; where missing .meta file rebuilds are
spotted.
--- Technical details follow ---
Several attempts to deal with this problem of multi-output targets, with and
without META MODE, were explained in r241298, r294878, and r301285.
The general problem is with multi-output targets such as:
foo.c foo.h:
touch foo.c foo.h
foo.c foo.h:
touch foo.c
touch foo.h
foo.c foo.h: foo.in
./generator ${.ALLSRC}
This pattern is problematic in jobs mode as both files end up being
built concurrently and leads to races. With META MODE it is worse
as both targets end up rebuilding if they lack a .meta file. So the
generator is force built twice even though it is only needed once.
There are also problems in that 'make foo.h' may be ran before 'make foo.c';
The order of make generating the targets is not guaranteed.
An older attempted workaround to this (discussed in r294878) was:
foo.h: foo.c
foo.c: foo.in
./generator ${.ALLSRC}
This appears fine except that if foo.h is missing and foo.c exists then
foo.h will never be regenerated. This pattern is close to the solution
in this commit though:
foo.h: foo.c .NOMETA
.if !exists(foo.h)
foo.c: .PHONY .META
.endif
foo.c: foo.in
./generator ${.ALLSRC}
There's 2 differences here:
1. foo.h will never expect to have a .meta file since the foo.c target
will generate both and own the .meta file.
2. If foo.h does not exist then it needs to force foo.c to be rebuilt
with .PHONY. That normally disables META MODE though so .META is
given to tell bmake we do really expect a .meta file.
This pattern cannot work with implicit suffix rules since the .c and .h files
may be generated at different times (buildincludes vs depend/all).
Sponsored by: Dell EMC
MFC after: 2 weeks
This is still targeting bin/sh cyclic dependency issues. Only apply
guessed dependencies that are explicitly set for an object (which
gnu/lib/cc/cc_tools needs) and if no custom target exists with its
own dependencies.
This was manifesting as a missing yacc.h in usr.bin/mkesdb_static when
built without -j (or -B). No actual yacc.h dependency ordering was
defined but with -j it got lucky and built fine.
Before r349061 the behavior was different for META_MODE but that logic
difference isn't needed.
X-MFC-With: r349061
Sponsored by: DellEMC
Default to tracking .depend.* for OBJS rather than SRCS.
This helps cover some special case builds like gnu/lib/csu which
do more of a PROGS-like thing with bsd.prog.mk.
It is possible this causes out-of-tree Makefiles to have problems if they use
this pattern:
foo.o: foo.c
${CC} -o ${.TARGET} ${.ALLSRC}
This may cause multiple source files to be compiled due to finding the
'foo.o: foo.c' dependency both in the Makefile at the .depend file. Or
it may try compiling headers. This can be worked around by either of these:
foo.o: foo.c
${CC} -o ${.TARGET} ${.ALLSRC:N*.h:[1]}
Or
foo.o: foo.c
${CC} -o ${.TARGET} ${.CURDIR}/foo.c
In the latter case the ${.CURDIR} may need to be a different path. The
first case covers automatically using .PATH.
Sponsored by: DellEMC
This is in the case of not having any .depend.foo.o yet. Don't force add *.h
as a dependency for those. They are built in beforebuild already when in
SRCS/DPSRCS.
This change allows custom rules, like in bin/sh/Makefile for mksyntax, to not
have cyclic dependency problems when connected to the .depend.* handling.
This is purposely not copied to sys/conf/kern.post.mk as it handles
generating headers slightly differently.
MFC after: 2 weeks
Sponsored by: DellEMC
Rtld is not compatible with SSP, and since we link libc_pic.a to rtld
to have the basic support like memory and string copy functions, we
have to both carefully limit libc use, and to provide the ssp support
shims. This change makes the libc use in rtld more straighforward but
still limited, and allows to remove the shims, to be done in the next
commit.
Submitted by: Luis Pires
Reviewed by: bdrewery, brooks
Differential revision: https://reviews.freebsd.org/D15283
The cleanobj target will not really remove the OBJDIR in this case,
it will only remove the OBJDIR if only clean targets are ran.
Sponsored by: Dell EMC Isilon
Some makefiles do reachover builds.
In some cases it is convenient to list subdirs of the distribution
in SRCS.
It is not very convenient, or always even desirable to have corresponding
subdirs in .OBJDIR, so OBJS_SRCS_FILTER allows the makefile to choose.
The default value 'R' matches existing practice.
But a makefile can set OBJS_SRCS_FILTER= T (the R gets added by
bsd.init.mk) to avoid the need for subdirs in .OBJDIR
Differential Revision: https://reviews.freebsd.org/D12218
Reviewed by: bdrewery
- Rename _SKIP_READ_DEPEND to _SKIP_DEPEND since it also avoids writing.
- This now uses .NOMETA to avoid reading any .meta files related to
DEPENDOBJS. Objects not in OBJS/DEPENDOBJS may still have their .meta
files read in if they are in the dependency graph.
- This also avoids statting .meta and .depend files in the META_MODE +
-DNO_FILEMON case.
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
This likely broke completely with r308599.
Apply the same fix for 'make destroy' which is a DIRDEPS_BUILD thing.
PR: 219819
Reported by: trasz
MFC after: 3 days
Sponsored by: Dell EMC Isilon
By default bmake does not allow meta mode to work unless an OBJDIR is
present. It allows this if curdirok= is set with a value not starting
with [0NnFf], but usually it is "yes".
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
_SKIP_BUILD will be set when nothing is expected to be built. This can
be used to optimize some tree-walks and operations which don't need to
load dependency files or generate dependencies via beforebuild-style
hacks.
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
build can break when different source files create the same target
files (case-insensitivity speaking). This is the case for object
files compiled with -fpic and shared libraries. The former uses
an extension of ".So", and the latter an extension ".so". Rename
shared object files from *.So to *.pico to match what NetBSD does.
See also r305855
MFC after: 1 month
Sponsored by: Bracket Computing
Differential Revision: https://reviews.freebsd.org/D7906
- For FAST_DEPEND, properly apply the -M flags when compiling by
enduring that the condition also has the s,/,_,g filter applied to it
first.
- For FAST_DEPEND, properly read from the filtered filename.
- For META_MODE, it needs to read from a full-pathed and s,/,_,g replaced
filename based on bmake's meta_name() function which names the .meta
file for SRCS with '/' in them.
This support has not been extended to the kernel build yet but may be in the
future.
MFC after: 2 weeks
Reported by: dim
Sponsored by: EMC / Isilon Storage Division
An optimization is in place to skip reading the .depend.* files with
'make install'. This was too strong and broke 'make all install' and
'make foo.o foo install'. Now only skip reading the dependency files
if all make targets ran are install targets.
The problem comes about because headers are only added in as a guessed
dependency if .depend.* files do not yet exist. If they do exist, even
if being skipped from being read, then the header dependencies are not
applied. This applies to all #included files, and not just headers.
Reported by: kib
MFC after: 1 day
Sponsored by: EMC / Isilon Storage Division
The DPADD data in .depend will be redundant with what is in the .meta file.
Also extend NO_EXTRADEPEND support to bsd.prog.mk.
Approved by: re (blanket, META_MODE)
Sponsored by: EMC / Isilon Storage Division
This is a follow-up to r300343.
This is important for the OBJS_DEPEND_GUESS usage in
gnu/usr.bin/cc/cc_tools.
See comments for more details.
Approved by: re (implicit)
Sponsored by: EMC / Isilon Storage Division
This is important to allow a Makefile to override OBJS_DEPEND_GUESS for
handling in META_MODE when its depend files are missing.
Approved by: re (implicit)
Sponsored by: EMC / Isilon Storage Division
The first file in these lists will generate everything else so only
it should be getting a .meta file. With bmake's missing=yes meta
feature these would otherwise cause a rebuild without the
.NOMETA hint.
Sponsored by: EMC / Isilon Storage Division
This breaks cross-building with WITH_META_MODE since it will rebuild
'build-tools' during the 'everything' phase.
A more proper fix is coming to bmake to implicitly require .META unless
.NOMETA (and other restrictions) are in place.
Adding .META to targets-to-build will ensure that they will rebuild if there
is no .meta file.
Adding it to all SUFFIXES and objects ensures that at least objects will
rebuild if there is no .meta file.
This will be reverted if bmake's behavior changes to rebuild on missing .meta
files.
Sponsored by: EMC / Isilon Storage Division
This is the same problem as r290629. With META_MODE we do not generate
.depend files, so there is no proper dependency to lookup. Guessed
dependencies must be used. If this proves to be a problem then we will
have to generate and use .depend files even with META_MODE.
Sponsored by: EMC / Isilon Storage Division
Otherwise the build command changes every build. META_MODE will
only remove it if something changes to warrant rebuilding
the file.
Sponsored by: EMC / Isilon Storage Division
The inclusion of .MAKE.DEPENDFILE (.depend) has special logic in make
to ignore stale/missing dependencies. bmake 20160220 added a '.dinclude'
directive that uses the special logic for .depend when including the file.
This fixes a build error when a file is moved or deleted that exists in a
.depend.OBJ file. This happened in r292782 when sha512c.c "moved" and an
incremental build of lib/libmd would fail with:
make: don't know how to make /usr/src/lib/libcrypt/../libmd/sha512c.c. Stop
Now this will just be seen as a stale dependency and cause a rebuild:
make: /usr/obj/usr/src/lib/libmd/.depend.sha512c.o, 13: ignoring stale .depend for /usr/src/lib/libcrypt/../libmd/sha512c.c
--- sha512c.o ---
...
This rebuild will only be done once since the .depend.sha512c.o will
be updated on the build with the -MF flags.
This also removes -MP being passed for the .depend.OBJ generation (which
would create fake targets for system headers) since the logic is no
longer needed to protect from missing files.
Sponsored by: EMC / Isilon Storage Division
This allows 'make analyze' or 'make OBJ.clang-analyzer' to run the
Clang static analyzer and present results on stdout.
Obtained from: NetBSD (CVS Rev. 1.3)
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5449
The OBJS_DEPEND_GUESS mechanism required moving the bsd.dep.mk inclusion
to after the checks, but left DEPENDFILE not-yet-set. Move it to
bsd.own.mk to resolve this.
Pointyhat to: bdrewery
Reported by: antoine (ports failures)
Sponsored by: EMC / Isilon Storage Division
This will generate dependencies rather than depending on the previous behavior
of depending on the guessed OBJS: *.h dependecies or a user running
'make depend'.
Experimentation showed that depending only on headers was not enough and
prone to .ORDER errors. Downstream users may also have added
dependencies into beforedepend or afterdepend targets. The safest way to
ensure dependencies are generated before build is to run 'make depend'
beforehand rather than just depending on DPSRCS+SRCS.
Note that the OBJS_DEPEND_GUESS mechanism (a.k.a .if !exists(.depend) then
foo.o: *.h) is still useful as it improves incremental builds with missing
.depend.* files and allows 'make foo.o' to usually work, while this
'beforebuild: depend' ensures that the build will always find all dependencies.
The 'make foo.o' case has no means of a 'beforebuild' hook.
This also removes several hacks in the DIRDEPS_BUILD:
- NO_INSTALL_INCLUDES is no longer needed as it mostly was to work around
.ORDER problems with building the needed headers early.
- DIRDEPS_BUILD: It is no longer necesarry to track "local dependencies" in
Makefile.depend.
These were only in Makefile.depend for 'clean builds' since nothing would
generate the files due to skipping 'make depend' and early dependency
bugs that have been fixed, such as adding headers into SRCS for the
OBJS_DEPEND_GUESS mechanism. Normally if a .depend file does not exist then
a dependency is added by bsd.lib.mk/bsd.prog.mk from OBJS: *.h. However,
meta.autodep.mk creates a .depend file from created meta files and inserts
that into Makefile.depend. It also only tracks *.[ch] files though which can
miss some dependencies that are hooked into 'make depend'. This .depend
that is created then breaks incremental builds due to the !exists(.depend)
checks for OBJS_DEPEND_GUESS. The goal was to skip 'make depend' yet it only
really works the first time. After that files are not generated as expected,
which r288966 tried to address but was using buildfiles: rather than
beforebuild: and was reverted in r291725. As noted previously,
depending only on headers in beforebuild: would create .ORDER errors
in some cases.
meta.autodep.mk is still used to generate Makefile.depend though via:
gendirdeps: Makefile.depend
.END: gendirdeps
This commit allows removing all of the "local dependencies" in
Makefile.depend which cuts down on churn and removes some of the
arch-dependent Makefile.depend files.
The "local dependencies" were also problematic for bootstrapping.
Sponsored by: EMC / Isilon Storage Division
FAST_DEPEND is intended to be the "skip 'make depend' and mkdep"
feature. Since DIRDEPS_BUILD does this already with some of its own
hacks, and filemon doesn't need this, and nofilemon does, teach it how
to handle each of these cases.
In meta+filemon mode filemon will handle dependencies itself via the
meta mode logic in bmake. We still want to set MK_FAST_DEPEND=yes to
enable some logic that indicates that 'make depend' is skipped in the
traditional sense. The actual .depend.* files will be skipped.
When nofilemon is set though we still need to track and generate dependencies.
Sponsored by: EMC / Isilon Storage Division
The .depend file will still be generated if _EXTRADEPEND is used. The target
is kept with a dependency on DPSRCS though so that 'make depend' will generate
all files.
Sponsored by: EMC / Isilon Storage Division
Rather than depend on .depend not existing, check the actual
.depend.OBJ file that will be used for that object. If it doesn't
exist then use the guessed dependencies.
FAST_DEPEND may never have a .depend file. Not having one means all of the
previous logic would over-depend all object files on all headers which is not
what we wanted. It also means that if a .depend is generated before a build
is done for _EXTRADEPEND (such as for PROG or LIB) then all of these
dependencies would not be used since the .depend wasn't generated from mkdep
and the real .depend.* files are not generated until the build.
Sponsored by: EMC / Isilon Storage Division
cleandepend should always remove CLEANDEPEND* if they are not empty,
but bsd.dep.mk should not add the tags entries unless SRCS is defined
as it did before. The .depend file itself it still always removed
to avoid accidentally keeping a stale one around as done in r295666.
Sponsored by: EMC / Isilon Storage Division
This will speed up some tree-walks with FAST_DEPEND which otherwise
would include length(SRCS) .depend files.
This also uses a trick suggested by sjg@ to still read them in when
specifying _V_READ_DEPEND=1 in the env/make args.
Sponsored by: EMC / Isilon Storage Division
