Add a wrapping script to use ATF to run tests written with Googletest
one by one. This helps locating and tracking the failing case in CI easier.
This is a temporarily solution while Googletest support in Kyua is developing.
We will revert this once Kyua+Googletest integration is ready.
Reviewed by: emaste
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D25896
googletest.test.mk is a rudimentary wrapper around the plain test interface
(for now), which only supports C++ programs, specified by the `GTESTS`
variable.
In the future, kyua will support gtests in a more native manner.
after r298107
Summary of changes:
- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)
Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.
MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division
- bsd.progs.mk is safe to include regardless of PROGS/PROGS_CXX/SCRIPTS
being set.
- bsd.progs.mk includes bsd.prog.mk always and will bring in
bsd.files.mk and bsd.obj.mk.
- DIRDEPS_BUILD: There's no need for _SKIP_BUILD or _SKIP_STAGING as the
PROGS were implicitly being built by the staging dependency anyway. This
was also preventing other objects from building if they were not part of
the staging sets.
- DIRDEPS_BUILD: This fixes PROGS without bsd.test.mk not staging.
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.
`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.
Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.
Document `make check` and `make checkworld` in build(7).
Other minor changes:
- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
`make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
not found
The MFC is [partly] contingent on other build related changes being MFCed.
Differential Revision: https://reviews.freebsd.org/D4406
MFC after: 2 months
X-MFC to: stable/10
Relnotes: yes
Reviewed by: bdrewery, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division
When run from bin/ls/tests, for example, the value of TESTSDIR would be
${TESTSBASE}/${RELDIR:H} -> /usr/tests/bin/ls/tests/.. ->
/usr/tests/bin/ls
Document the new behavior in bsd.README.
While here, also document TESTSBASE
Relnotes: yes
Differential Revision: D1022
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
This mostly fixes an interaction with bsd.test.mk with PROGS and SCRIPTS.
This was most notable with 'make clean' and 'make install', which r281055
and r272055 attempted to address but were inadequate.
It also addresses similar issues in bsd.progs.mk when not using bsd.test.mk.
This also fixes cases of NOT running commands in the parent when using
bsd.progs.mk:
- 'make clean' was not run for the main process for Makefiles which had both
FILES and SUBDIR but no PROGS or SCRIPTS. This usually was just a
leftover Kyuafile.auto. One such example is usr.bin/bmake/tests/sysmk/t1/2.
- 'make obj' was not running in the current directory with bsd.test.mk due
to early inclusion of bsd.subdir.mk. This was not really a problem due to
the SUBDIRS using 'mkdir -p' for their objdirs.
There were subtle bugs causing this wrong behavior:
1. bsd.progs.mk needs to set SCRIPTS to empty when recursing to avoid
the sub-makes from installing, cleaning or building the SCRIPTS;
only the parent make should be doing this. r281055 effectively did
the same but wasn't enough.
2. CLEANFILES may contain (especially from *.test.mk) files which only
the parent should clean, such as from FILES and SCRIPTS. To resolve
sub-makes also cleaning these, reset CLEANFILES and CLEANDIRS in the
children before including bsd.prog.mk. A tempting alternative would be
to only handle CLEANFILES in the parent but then the child bsd.prog.mk
CLEANFILES of per-PROGS wouldn't be setup.
3. bsd.subdir.mk was included too soon in bsd.test.mk. It needs to be
included after bsd.prog.mk as the SCRIPTS logic is short-circuitted if
'install:' is already defined (which bsd.subdir.mk does). There is
actually no need to include bsd.subdir.mk from bsd.test.mk as bsd.prog.mk
and bsd.obj.mk will do so in the proper order. The description in r257095
covers this for FILES and was fixed differently, though changing the
handling of target(install) in bsd.prog.mk may make sense after more
research.
4. bsd.progs.mk had extra logic to handle recursing SCRIPTS if PROGS was
empty, which isn't its business to be doing. SCRIPTS is handled fine
by bsd.prog.mk. This mostly reverts and reworks the fix in r259209 and
partially reverts r272055.
5. bsd.progs.mk has no need to depend 'all:' on SCRIPTS and FILES. These
are handled by bsd.prog.mk/bsd.files.mk fine. This also partially reverts
r272055.
6. bsd.progs.mk was not drop-in safe for bsd.prog.mk. Move the PROGS
check from r273186 to allow it to be used safely.
Specific tested cases:
SCRIPTS:no PROGS:no FILES:yes SUBDIR:yes
usr.bin/bmake/tests/sysmk/t1/2
SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no
usr.bin/bmake/tests/sysmk/t1/2/1
SCRIPTS:yes PROGS:yes FILES:yes SUBDIR:yes
lib/libthr/tests
SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no
usr.bin/yacc/tests
libexec/atf/atf-sh/tests
A full buildworld/installworld/clean comparison with mtree was also done.
The only relevant difference was the new fixed behavior of removing
Kyuafile.auto from the objdir in 'clean'.
Converting SCRIPTS to be a special case FILES group will make this less
fragile and is being explored.
One known remaining issue is 'cleandepend' removing the tags files for
every recursive call.
Note that the 'make clean' command runs for the CURDIR last, which can make
it appear to run multiple times when cleaning in tests/, but each command is
for a SUBDIR returning up the chain. This is purely bsd.subdir.mk behavior.
PR: 191055
PR: 191955
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
expression (:M) is empty, not the not matched (:N) is empty. The former case
means we have not found the TEST_SUBDIR value in SUBDIR
Reported by: rodrigc
X-MFC with: r273803
Pointyhat to: me (did not use a clean install root)
Sponsored by: EMC / Isilon Storage Division
appending the TESTS_SUBDIRS variable to SUBDIR
Duplicate directory entries can cause unexpected side effects, like
installing the same files multiple times. This can be easily
reproduced via the following testcase prior to this commit:
SUBDIR= dir
TESTS_SUBDIRS+= dir
.include <bsd.test.mk>
Sponsored by: EMC / Isilon Storage Division
Force all the contents of /usr/tests to go into a separate distribution
file so that users of binary releases can easily choose to not install it.
To make this possible, we need two fixes:
- bsd.subdir.mk needs to properly honor NO_SUBDIR in all cases so that we
do not recurse into 'tests' subdirectories when we needn't. Otherwise,
we end up with some Kyuafiles in base.txz.
- etc/Makefile needs to skip installing tests in its 'distribute' target
so that a Kyuafile doesn't leak into base.txz.
Approved by: gjb
When FILES is defined in a Makefile that _also_ includes bsd.subdir.mk, the
build of the files (if any) was not properly triggered during the build
stage. This was because bsd.files.mk did not define the buildfiles target
if it was already defined... and bsd.subdir.mk defined this target on its
own, thus causing a conflict.
Fix this by unconditionally defining buildfiles from bsd.files.mk; this is
safe because nothing else in the tree needs to redefine this and because the
target itself contains no commands: all it does is define dependencies.
Also ensure that bsd.files.mk is always pulled in by bsd.test.mk regardless
of what bsd.prog.mk does.
These fixes allow "make installworld" to run cleanly on a system with
read-only src and obj trees.
This is "make tinderbox" clean.
Reviewed by: imp
Obtained from: jilles
The new suite.test.mk file contains all the logic needed to install test
programs under /usr/tests/ and to support Kyua as the run-time engine.
This file is included by default by bsd.test.mk so Makefiles do not need
to care about its existence.
Specific Makefiles can define NOT_FOR_TEST_SUITE to indicate that whatever
test programs they are building are not supposed to be installed under
/usr/tests/ nor run by Kyua. (The effect of passing this setting is that
suite.test.mk is simply not included.)
NOT_FOR_TEST_SUITE should never be used by Makefiles in the base system.
This functionality is provided so that third-parties can hook in their
own test code, with different semantics, if they wish. This was asked
for by sjg@.
Change {atf,plain,tap}.test.mk to be internal implementation details of
bsd.test.mk. Makefiles that build tests should now only include bsd.test.mk
and declaratively specify what they want to build, without worrying about
the internal implementation of the mk files.
The reason for this change is to permit building test programs of different
interfaces from a single directory, which is something I had a need for
while porting tests over from src/tools/regression/.
Additionally, this change makes it possible to perform some other requested
changes to bsd.test.mk in an easier manner. Coming soon.
Introduce a new, per-test-program TEST_METADATA.<program> variable that
contains a list of key/value paris describing metadata properties for
that test program. These properties are later written into the
auto-generated Kyuafile when using the KYUAFILE=auto functionality.
This is to avoid having to supply hand-crafted Kyuafiles when the needs
for metadata overrides are trivial.
While doing this, and because I am documenting TEST_METADATA, take the
chance to document the TEST_INTERFACE setting as well.
MFC after: 5 days
When generating a Kyuafile in the KYUAFILE=auto case, use a filename
that is unlikely to clash with the filename used by explicitly-provided
Kyuafiles.
This allows a Makefile to set KYUAFILE=yes and provide a Kyuafile in
the same directory when such Makefile was previously relying on
KYUAFILE=auto.
Fixes issues with new Kyuafiles not being picked up in NO_CLEAN builds
(although manual intervention is required once, unfortunately, as
described in UPDATING).
Reviewed by: sjg
MFC after: 1 week
We need to be able to reference the value of TESTSBASE without requiring
the inclusion of bsd.test.mk (e.g. in etc/Makefile), so move its definition
to the more generic bsd.own.mk.
Submitted by: Julio Merino jmmv google.com
Reviewed by: sjg
MFC after: 2 weeks
If kyua exists in KYUA_PREFIX, the test target is automatically
defined to use it for the execution of test programs.
Submitted by: Julio Merino jmmv google.com
MFC after: 2 weeks
If atf-run exists in ATF_PREFIX and if ALLOW_DEPRECATED_ATF_TOOLS has
been set to yes, the test target is automatically defined to use it
for the execution of test programs.
Submitted by: Julio Merino jmmv google.com
MFC after: 2 weeks
These files are generated from bsd.test.mk because kyua is able to run
test programs implemented using different libraries/frameworks. In
order to make this possible, this change also extends the various
*.test.mk file to explicitly indicate the interface of every test
program.
Submitted by: Julio Merino jmmv google.com
MFC after: 2 weeks
These are only used by the deprecated atf-run and atf-report tools.
Generating them is easy and provides a mechanism for people to
experiment with these tools if they wish.
But, because these tools and files are deprecated, doing this only
happens if the user has explicitly set ALLOW_DEPRECATED_ATF_TOOLS
to yes.
Submitted by: Julio Merino jmmv google.com
MFC after: 2 weeks
This change introduces a new plain.test.mk file that provides the build
infrastructure to build test programs that don't use any framework.
Most of the code previously in bsd.test.mk moves to plain.test.mk and
atf.test.mk is extended with the missing pieces.
In doing so, this change pushes all test program building logic to the
various *.test.mk files instead of trying to reuse some tiny bits.
In fact, this attempt to reuse some definitions makes the code harder
to read and harder to extend.
The clear benefit of this is that the interface of bsd.test.mk is now
clearly delimited.
Submitted by: Julio Merino jmmv google.com
MFC after: 2 weeks
