freebsd-skq/tests
ngie 631df11d04 Integrate capsicum-test into the FreeBSD test suite
This change takes capsicum-test from upstream and applies some local changes to make the
tests work on FreeBSD when executed via Kyua.

The local modifications are as follows:
1. Make `OpenatTest.WithFlag` pass with the new dot-dot lookup behavior in FreeBSD 12.x+.
2. capsicum-test references a set of helper binaries: `mini-me`, `mini-me.noexec`, and
   `mini-me.setuid`, as part of the execve/fexecve tests, via execve, fexecve, and open.
   It achieves this upstream by assuming `mini-me*` is in the current directory, however,
   in order for Kyua to execute `capsicum-test`, it needs to provide a full path to
   `mini-me*`. In order to achieve this, I made `capsicum-test` cache the executable's
   path from argv[0] in main(..) and use the cached value to compute the path to
   `mini-me*` as part of the execve/fexecve testcases.
3. The capsicum-test test suite assumes that it's always being run on CAPABILITIES enabled
   kernels. However, there's a chance that the test will be run on a host without a
   CAPABILITIES enabled kernel, so we must check for the support before running the tests.
   The way to achieve this is to add the relevant `feature_present("security_capabilities")`
   check to SetupEnvironment::SetUp() and skip the tests when the support is not available.
   While here, add a check for `kern.trap_enotcap` being enabled. As noted by markj@ in
   https://github.com/google/capsicum-test/issues/23, this sysctl being enabled can trigger
   non-deterministic failures. Therefore, the tests should be skipped if this sysctl is
   enabled.

All local changes have been submitted to the capsicum-test project
(https://github.com/google/capsicum-test) and are in various stages of review.
Please see the following pull requests for more details:
1. https://github.com/google/capsicum-test/pull/35
2. https://github.com/google/capsicum-test/pull/41
3. https://github.com/google/capsicum-test/pull/42

Reviewed by:	asomers
Discussed with:	emaste, markj
Approved by:	emaste (mentor)
MFC after:	2 months
Differential Revision: https://reviews.freebsd.org/D19758
2019-04-01 21:24:50 +00:00
..
etc Add supporting changes for Add limited sandbox capability to "make check" 2017-08-14 19:21:37 +00:00
freebsd_test_suite Fix sys/opencrypto/blake2_test when kern.cryptodevallowsoft=0 2018-08-16 23:49:56 +00:00
sys Integrate capsicum-test into the FreeBSD test suite 2019-04-01 21:24:50 +00:00
Kyuafile
Makefile Use MK_CHECK_USE_SANDBOX in tests/..., to deal with the fact that 2017-08-02 22:24:08 +00:00
Makefile.depend
Makefile.inc0 Use bsd.opts.mk, not src.opts.mk 2017-08-03 00:35:35 +00:00
README

src/tests: The FreeBSD test suite
=================================

To run the FreeBSD test suite:
(1)  Make sure that kyua is installed:
       pkg install kyua
(2)  To run the tests:
       kyua test -k /usr/tests/Kyuafile
(3)  To see the test results:
       kyua report  

For further information on using the test suite, read tests(7):
       man tests

Description of FreeBSD test suite
=================================
The build of the test suite is organized in the following manner:

* The build of all test artifacts is protected by the MK_TESTS knob.
  The user can disable these with the WITHOUT_TESTS setting in
  src.conf(5).

* The goal for /usr/tests/ (the installed test programs) is to follow
  the same hierarchy as /usr/src/ wherever possible, which in turn drives
  several of the design decisions described below.  This simplifies the
  discoverability of tests.  We want a mapping such as:

    /usr/src/bin/cp/      -> /usr/tests/bin/cp/
    /usr/src/lib/libc/    -> /usr/tests/lib/libc/
    /usr/src/usr.bin/cut/ -> /usr/tests/usr.bin/cut/
    ... and many more ...

* Test programs for specific utilities and libraries are located next
  to the source code of such programs.  For example, the tests for the
  src/lib/libcrypt/ library live in src/lib/libcrypt/tests/.  The tests/
  subdirectory is optional and should, in general, be avoided.

* The src/tests/ hierarchy (this directory) provides generic test
  infrastructure and glue code to join all test programs together into
  a single test suite definition.

* The src/tests/ hierarchy also includes cross-functional test programs:
  i.e. test programs that cover more than a single utility or library
  and thus don't fit anywhere else in the tree.  Consider this to follow
  the same rationale as src/share/man/: this directory contains generic
  manual pages while the manual pages that are specific to individual
  tools or libraries live next to the source code.

In order to keep the src/tests/ hierarchy decoupled from the actual test
programs being installed --which is a worthy goal because it simplifies
the addition of new test programs and simplifies the maintenance of the
tree-- the top-level Kyuafile does not know which subdirectories may
exist upfront.  Instead, such Kyuafile automatically detects, at
run-time, which */Kyuafile files exist and uses those directly.

Similarly, every directory in src/ that wants to install a Kyuafile to
just recurse into other subdirectories reuses this Kyuafile with
auto-discovery features.  As an example, take a look at src/lib/tests/
whose sole purpose is to install a Kyuafile into /usr/tests/lib/.
The goal in this specific case is for /usr/tests/lib/ to be generated
entirely from src/lib/.

-- 
$FreeBSD$