== Rationale ==
r295380 introduced "make check" and consolidated means for running
test code in an attempt to simplify running tests. One could either
install files/libraries/programs and run "make check", or run "make check"
with an explicit CHECKDIR, e.g., `make check CHECKDIR=$(make -V.OBJDIR)``.
One criticism that was received is that "make check" should be run with
the intent of making dev->test->commit easier, which means that the target
audience's workflow should be developers. One developer pattern available
in other opensource projects is to run test code from a developer sandbox,
instead of installing to a system.
== Method ==
This approach is slightly different from the standard approach, in the sense
that it builds and installs into a deterministic directory under .OBJDIR (as I call it,
the "sandbox"), then runs "make check" against that. In the event the test
run is successful, the deterministic directory is removed to save space.
== Approach ==
bsd.lib.mk, bsd.prog.mk:
To support this functionality, a new variable `HAS_TESTS` is being added.
HAS_TESTS enables appropriate behavior with bsd.lib.mk and bsd.prog.mk, as
follows:
- Add "make check" as an available target from the directory.
- Pass down appropriate variables via ${TESTS_ENV}, i.e.,
${TESTS_LD_LIBRARY_PATH} and ${TESTS_PATH}.
One should add "HAS_TESTS" to directories containing tests in them, e.g. from
bin/sh/Makefile,
HAS_TESTS=
SUBDIR.${MK_TESTS}+= tests
HAS_TESTS doesn't automatically add the tests subdirectory for flexibility
reasons.
bsd.opts.mk, src.opts.mk:
- The knob ${MK_MAKE_CHECK_USE_SANDBOX} has been added, both to explicitly
direct (internally) when to set a deterministic ${DESTDIR} and to also allow
users to disable this behavior globally, i.e., via src.conf.
- MK_TESTS has been promoted from src.opts.mk to bsd.opts.mk to leverage
syntactic sugar for having MK_TESTS be a dependency for
MK_MAKE_CHECK_USE_SANDBOX, but to also ensure that src.opts.mk isn't required
to use suite.test.mk (which is a dependency of bsd.test.mk).
suite.test.mk:
- beforecheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified
from a no-op to:
-- Build.
-- Run "make hierarchy" on the sandbox dir.
-- Install the tests/files to the sandbox dir.
- aftercheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified
from a no-op to:
-- Remove the sandbox dir.
Again, because the dependency order set in bsd.test.mk is
beforecheck -> check -> aftercheck, "make check" will not be run unless
"beforecheck" completes successfully, and "aftercheck" will not be run unless
"beforecheck" and "check" complete successfully.
== Caveats ==
- This target must either be run with MK_INSTALL_AS_USER or as root. Otherwise
it will fail when running "make install" as the default user/group for many
makefiles when calling INSTALL is root/wheel.
- This target must be run from a suitable top-level directory. For example,
running tests from `tests/sys/fs/tmpfs` won't work, but `tests/sys/fs` will,
because `tests/sys/fs/tmpfs` relies on files installed by `tests/sys/fs`.
- Running MK_INSTALL_AS_USER may introduce determinism issues. However, using
it could identify deficiences in tests in terms of needing to be run as
root, which are not properly articulated in the test requirements.
- The doesn't negate the need for running "make installworld" and
"make checkworld", etc. Again, this just is intended to simplify the
dev->test->commit workflow.
== Cleanup done ==
- CHECKDIR is removed; one can use "MK_MAKE_CHECK_USE_SANDBOX=no" to enable
"legacy" (r295380) behavior.
MFC after: 2 months
Relnotes: yes (CHECKDIR removed; "make check" behavior changed)
Requested by: jhb
Reviewed by: arch (silence), testing (silence)
Differential Revision: D11905
The latter set of manpages directly consume bsd.man.mk, so the bsd.man.mk
behavior should be the source of truth for underlying behavior, whereas
the other manpage fragment descriptions should document how they tweak
the variable behavior, if at all (bsd.prog.mk does tweak the default
value, as noted in its description)
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Running smilint against MIB definitions is useful in finding
functional problems with MIB definitions/descriptions.
This is inspired by the smilint targets defined in
usr.sbin/bsnmpd/modules/{snmp_hostres,snmp_mibII}/Makefile
Document all of the variables that are involved in running the
smilint target, as well as all of the prerequisites to running
it.
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D9099
Prefix with dashes (unordered list) and put one variable on each
line (to avoid future conflicts)
Done via the following one-liner:
> sh -c 'for i in $(make -C tests/sys/aio PROG=foo -VPROG_VARS:O); do printf "\t\t- $i\n"; done'
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.
`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.
Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.
Document `make check` and `make checkworld` in build(7).
Other minor changes:
- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
`make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
not found
The MFC is [partly] contingent on other build related changes being MFCed.
Differential Revision: https://reviews.freebsd.org/D4406
MFC after: 2 months
X-MFC to: stable/10
Relnotes: yes
Reviewed by: bdrewery, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division
When run from bin/ls/tests, for example, the value of TESTSDIR would be
${TESTSBASE}/${RELDIR:H} -> /usr/tests/bin/ls/tests/.. ->
/usr/tests/bin/ls
Document the new behavior in bsd.README.
While here, also document TESTSBASE
Relnotes: yes
Differential Revision: D1022
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
and finish the job. ncurses is now the only Makefile in the tree that
uses it since it wasn't a simple mechanical change, and will be
addressed in a future commit.
of files other than the actual libraries.
Use LIBRARIES_ONLY to supress the inclusion of files in the lib32
distribution that are duplicates of files in base.
Sponsored by: DARPA, AFRL
Reviewed by: emaste
for both of them use different, and presumably wrong, variables in the
example. They set LDFILES and SRCLIB respectively. I guess that's what
DPADD and LDADD were called first ...
use in-tree as well as for 3rd party modules. This file is more or less
what was in usr.sbin/bsnmpd/modules/Makefile.inc with some modifications
and omissions. Usage examples can be found under usr.sbin/bsnmpd/modules/*.
Idea by: phk
