freebsd-dev/tests
Eric van Gyzen 11ed0a95bf zfs tests: stop writing to arbitrary devices
TL;DR:  Three ZFS tests created ZFS pools on all unmounted devices listed
in /etc/fstab, corrupting their contents.  Stop that.

Imagine my surprise when the ESP on my main dev/test VM would "randomly"
become corrupted, making it unbootable.  Three tests collect various devices
from the system and try to add them to a test pool.  The test expects this
to fail because it _assumes_ these devices are in use and ZFS will correctly
reject the request.

My /etc/fstab has two entries for devices in /dev:

    /dev/gpt/swap0  none        swap    sw,trimonce,late
    /dev/gpt/esp0   /boot/efi   msdosfs rw,noauto

Note the `noauto` on the ESP.  In a remarkable example of irony, I chose
this because it should keep the ESP more protected from corruption;
in fact, mounting it would have protected it from this case.

The tests added all of these devices to a test pool in a _single command_,
expecting the command to fail.  The swap device was in use, so the command
correctly failed, but the ESP was added and therefore corrupted.  However,
since the command correctly failed, the test didn't notice the ESP problem.
If each device had been added with its own command, the test _might_ have
noticed that one of them incorrectly succeeded.  However, two of these
tests would not have noticed:

hotspare_create_001_neg was incorrectly specified as needing the Solaris
dumpadm command, so it was skipped.  _Some_ of the test needs that command,
but it checks for its presence and runs fine without it.

Due to bug 241070, zpool_add_005_pos was marked as an expected failure.
Due to the coarse level of integration with ATF, this test would still
"pass" even if it failed for the wrong reason.  I wrote bug 267554 to
reconsider the use of atf_expect_fail in these tests.

Let's further consider the use of various devices found around the system.
In addition to devices in /etc/fstab, the tests also used mounted devices
listed by the `mount` command.  If ZFS behaves correctly, it will refuse
to added mounted devices and swap devices to a pool.  However, these are
unit tests used by developers to ensure that ZFS still works after they
modify it, so it's reasonable to expect ZFS to do the _wrong_ thing
sometimes.  Using random host devices is unsafe.

Fix the root problem by using only the disks provided via the "disks"
variable in kyua.conf.  Use one to create a UFS file system and mount it.
Use another as a swap device.  Use a third as a dump device, but expect
it to fail due to bug 241070.

While I'm here:

Due to commit 6b6e2954dd, we can simply add a second dump device and
remove it in cleanup.  We no longer need to save, replace, and restore the
pre-existing dump device.

The cleanup_devices function used `camcontrol inquiry` to distinguish disks
from other devices, such as partitions.  That works fine for SCSI, but not
for ATA or VirtIO block.  Use `geom disk list` instead.

PR:		241070
PR:		267554
Reviewed by:	asomers
Sponsored by:	Dell Inc.
Differential Revision:	https://reviews.freebsd.org/D37257
2022-11-11 14:43:47 -06:00
..
atf_python tests: add routing tests for switching between same prefixes 2022-08-07 19:45:25 +00:00
etc Add supporting changes for Add limited sandbox capability to "make check" 2017-08-14 19:21:37 +00:00
freebsd_test_suite testing: move atf-pytest-wrapper to /usr/libexec 2022-06-26 13:25:47 +00:00
sys zfs tests: stop writing to arbitrary devices 2022-11-11 14:43:47 -06:00
__init__.py testing: Add basic atf support to pytest. 2022-06-25 19:25:15 +00:00
conftest.py testing: pass ATF vars to pytest via env instead of arguments. 2022-06-28 12:20:16 +00:00
Kyuafile
Makefile testing: move atf-pytest-wrapper to /usr/libexec 2022-06-26 13:25:47 +00:00
Makefile.depend DIRDEPS_BUILD: Connect MK_TESTS. 2016-03-09 22:46:01 +00:00
Makefile.inc0 Use bsd.opts.mk, not src.opts.mk 2017-08-03 00:35:35 +00:00
README Revert "wpa: Import wpa 2.10." 2022-01-18 08:10:33 -08:00

src/tests: The FreeBSD test suite
=================================

Usage of the FreeBSD test suite:
(1)  Run the tests:
       kyua test -k /usr/tests/Kyuafile
(2)  See the test results:
       kyua report

For further information on using the test suite, read tests(7):
       man tests

Description of FreeBSD test suite
=================================
The build of the test suite is organized in the following manner:

* The build of all test artifacts is protected by the MK_TESTS knob.
  The user can disable these with the WITHOUT_TESTS setting in
  src.conf(5).

* The goal for /usr/tests/ (the installed test programs) is to follow
  the same hierarchy as /usr/src/ wherever possible, which in turn drives
  several of the design decisions described below.  This simplifies the
  discoverability of tests.  We want a mapping such as:

    /usr/src/bin/cp/      -> /usr/tests/bin/cp/
    /usr/src/lib/libc/    -> /usr/tests/lib/libc/
    /usr/src/usr.bin/cut/ -> /usr/tests/usr.bin/cut/
    ... and many more ...

* Test programs for specific utilities and libraries are located next
  to the source code of such programs.  For example, the tests for the
  src/lib/libcrypt/ library live in src/lib/libcrypt/tests/.  The tests/
  subdirectory is optional and should, in general, be avoided.

* The src/tests/ hierarchy (this directory) provides generic test
  infrastructure and glue code to join all test programs together into
  a single test suite definition.

* The src/tests/ hierarchy also includes cross-functional test programs:
  i.e. test programs that cover more than a single utility or library
  and thus don't fit anywhere else in the tree.  Consider this to follow
  the same rationale as src/share/man/: this directory contains generic
  manual pages while the manual pages that are specific to individual
  tools or libraries live next to the source code.

In order to keep the src/tests/ hierarchy decoupled from the actual test
programs being installed --which is a worthy goal because it simplifies
the addition of new test programs and simplifies the maintenance of the
tree-- the top-level Kyuafile does not know which subdirectories may
exist upfront.  Instead, such Kyuafile automatically detects, at
run-time, which */Kyuafile files exist and uses those directly.

Similarly, every directory in src/ that wants to install a Kyuafile to
just recurse into other subdirectories reuses this Kyuafile with
auto-discovery features.  As an example, take a look at src/lib/tests/
whose sole purpose is to install a Kyuafile into /usr/tests/lib/.
The goal in this specific case is for /usr/tests/lib/ to be generated
entirely from src/lib/.

-- 
$FreeBSD$