freebsd-dev

History

Alex Richardson 53a535c1d8 Simplify the capsicum-test wrapper script Instead of running tests one-by-one with the shell wrapper we now run the full gtest testsuite twice (once as root, once as non root). This significantly speeds up running tests despite running them twice. This change also passes the missing -u flag to capsicum-test that caused test failures (https://bugs.freebsd.org/250178) Previously, running the testsuite with the wrapper script took ~3s per test on aarch64 QEMU, i.e. a total of almost 5 minutes. Now it takes 6 seconds to run all tests twice. Before: root@freebsd-aarch64:/usr/tests/sys/capsicum # /usr/bin/time kyua test functional 94/96 passed (2 failed) 309.97 real 58.46 user 244.31 sys After: root@freebsd-aarch64:/usr/tests/sys/capsicum # /usr/bin/time kyua test functional functional:test_root -> passed [2.659s] functional:test_unprivileged -> passed [2.391s] 2/2 passed (0 failed) 5.48 real 1.06 user 2.52 sys This overhead is caused by kyua + atf-sh spawning lots of additional processes and can be avoided by just running the googletest test binary. syscall seconds calls errors fork 39.810229456 1275 0 sigprocmask 13.546928736 572 0 i.e. 1275 processes spawned to run a single test. Test Plan: All tests pass with D28907. PR: 250178 Reviewed By: lwhsu Differential Revision: https://reviews.freebsd.org/D29014		2021-03-02 18:27:36 +00:00
..
etc	Add supporting changes for `Add limited sandbox capability to "make check"`	2017-08-14 19:21:37 +00:00
freebsd_test_suite	Fix sys/opencrypto/blake2_test when kern.cryptodevallowsoft=0	2018-08-16 23:49:56 +00:00
sys	Simplify the capsicum-test wrapper script	2021-03-02 18:27:36 +00:00
Kyuafile	Set up the /usr/tests hierarchy.	2013-10-25 05:25:19 +00:00
Makefile	Tag /usr/tests/local symlink with package=tests	2020-01-23 15:59:30 +00:00
Makefile.depend	DIRDEPS_BUILD: Connect MK_TESTS.	2016-03-09 22:46:01 +00:00
Makefile.inc0	Use bsd.opts.mk, not src.opts.mk	2017-08-03 00:35:35 +00:00
README	Do not recommend to install kyua with pkg in the tests README	2020-10-27 09:53:49 +00:00

README

src/tests: The FreeBSD test suite
=================================

Usage of the FreeBSD test suite:
(1)  Run the tests:
       kyua test -k /usr/tests/Kyuafile
(2)  See the test results:
       kyua report

For further information on using the test suite, read tests(7):
       man tests

Description of FreeBSD test suite
=================================
The build of the test suite is organized in the following manner:

* The build of all test artifacts is protected by the MK_TESTS knob.
  The user can disable these with the WITHOUT_TESTS setting in
  src.conf(5).

* The goal for /usr/tests/ (the installed test programs) is to follow
  the same hierarchy as /usr/src/ wherever possible, which in turn drives
  several of the design decisions described below.  This simplifies the
  discoverability of tests.  We want a mapping such as:

    /usr/src/bin/cp/      -> /usr/tests/bin/cp/
    /usr/src/lib/libc/    -> /usr/tests/lib/libc/
    /usr/src/usr.bin/cut/ -> /usr/tests/usr.bin/cut/
    ... and many more ...

* Test programs for specific utilities and libraries are located next
  to the source code of such programs.  For example, the tests for the
  src/lib/libcrypt/ library live in src/lib/libcrypt/tests/.  The tests/
  subdirectory is optional and should, in general, be avoided.

* The src/tests/ hierarchy (this directory) provides generic test
  infrastructure and glue code to join all test programs together into
  a single test suite definition.

* The src/tests/ hierarchy also includes cross-functional test programs:
  i.e. test programs that cover more than a single utility or library
  and thus don't fit anywhere else in the tree.  Consider this to follow
  the same rationale as src/share/man/: this directory contains generic
  manual pages while the manual pages that are specific to individual
  tools or libraries live next to the source code.

In order to keep the src/tests/ hierarchy decoupled from the actual test
programs being installed --which is a worthy goal because it simplifies
the addition of new test programs and simplifies the maintenance of the
tree-- the top-level Kyuafile does not know which subdirectories may
exist upfront.  Instead, such Kyuafile automatically detects, at
run-time, which */Kyuafile files exist and uses those directly.

Similarly, every directory in src/ that wants to install a Kyuafile to
just recurse into other subdirectories reuses this Kyuafile with
auto-discovery features.  As an example, take a look at src/lib/tests/
whose sole purpose is to install a Kyuafile into /usr/tests/lib/.
The goal in this specific case is for /usr/tests/lib/ to be generated
entirely from src/lib/.

-- 
$FreeBSD$