245 lines
7.5 KiB
Groff
245 lines
7.5 KiB
Groff
.\" $FreeBSD$
|
|
.\" $NetBSD: tests.kyua.7,v 1.2 2013/07/20 21:39:59 wiz Exp $
|
|
.\"
|
|
.\" Copyright (c) 2010 The NetBSD Foundation, Inc.
|
|
.\" All rights reserved.
|
|
.\"
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
.\" modification, are permitted provided that the following conditions
|
|
.\" are met:
|
|
.\" 1. Redistributions of source code must retain the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer.
|
|
.\" 2. Redistributions in binary form must reproduce the above copyright
|
|
.\" notice, this list of conditions and the following disclaimer in the
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
.\"
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
|
|
.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
|
|
.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
|
|
.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
|
|
.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
|
|
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
|
|
.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
|
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
|
|
.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
|
|
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
|
|
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
.\"
|
|
.Dd August 19, 2020
|
|
.Dt TESTS 7
|
|
.Os
|
|
.Sh NAME
|
|
.Nm tests
|
|
.Nd introduction to the
|
|
.Fx
|
|
Test Suite
|
|
.Sh DESCRIPTION
|
|
The
|
|
.Fx
|
|
Test Suite provides a collection of automated tests for two major purposes.
|
|
On one hand, the test suite aids
|
|
.Em developers
|
|
to detect bugs and regressions when they modify the source tree.
|
|
On the other hand, it allows
|
|
.Em end users
|
|
(and, in particular, system administrators) to verify that fresh installations
|
|
of the
|
|
.Fx
|
|
operating system behave correctly on their hardware platform and also to ensure
|
|
that the system does not suffer from regressions during regular operation and
|
|
maintenance.
|
|
.Pp
|
|
The
|
|
.Fx
|
|
Test Suite can be found in the
|
|
.Pa /usr/tests
|
|
hierarchy.
|
|
.Pp
|
|
This manual page describes how to run the test suite and how to configure
|
|
some of its optional features.
|
|
For information on writing the tests, see
|
|
.Xr atf 7 .
|
|
.Ss Installing the test suite
|
|
If the
|
|
.Pa /usr/tests
|
|
directory is missing, then you will have to enable the build of the test
|
|
suite, rebuild your system and install the results.
|
|
You can do so by setting
|
|
.Sq WITH_TESTS=yes
|
|
in your
|
|
.Pa /etc/src.conf
|
|
file (see
|
|
.Xr src.conf 5
|
|
for details)
|
|
and rebuilding the system as described in
|
|
.Xr build 7 .
|
|
.Ss When to run the tests?
|
|
Before diving into the details of how to run the test suite, here are some
|
|
scenarios in which you should run it:
|
|
.Bl -bullet -offset indent
|
|
.It
|
|
After a fresh installation of
|
|
.Fx
|
|
to ensure that the system works correctly on your hardware platform.
|
|
.It
|
|
After an upgrade of
|
|
.Fx
|
|
to a different version to ensure that the new code works well on your
|
|
hardware platform and that the upgrade did not introduce regressions in your
|
|
configuration.
|
|
.It
|
|
After modifying the source tree to detect any new bugs and/or regressions.
|
|
.It
|
|
Periodically, maybe from a
|
|
.Xr cron 8
|
|
job, to ensure that any changes to the system (such as the installation of
|
|
third-party packages or manual modifications to configuration files) do not
|
|
introduce unexpected failures.
|
|
.El
|
|
.Ss Running the tests
|
|
Use the following command to run the whole test suite:
|
|
.Bd -literal -offset indent
|
|
$ kyua test -k /usr/tests/Kyuafile
|
|
.Ed
|
|
.Pp
|
|
The above will iterate through all test programs in
|
|
.Pa /usr/tests
|
|
recursively, execute them, store their results and debugging data in Kyua's
|
|
database (by default in
|
|
.Pa ~/.kyua/store.db ) ,
|
|
and print a summary of the results.
|
|
This summary includes a brief count of all total tests run and how many of
|
|
them failed.
|
|
.Pp
|
|
It is possible to restrict which tests to run by providing their names in
|
|
the command line.
|
|
For example, this would execute the tests for the
|
|
.Xr cp 1
|
|
and
|
|
.Xr cut 1
|
|
utilities:
|
|
.Bd -literal -offset indent
|
|
$ kyua test -k /usr/tests/Kyuafile bin/cp usr.bin/cut
|
|
.Ed
|
|
.Ss Obtaining reports of the tests execution
|
|
Additional information about the test results can be retrieved
|
|
by using Kyua's various reporting commands.
|
|
For example, the following would print a plain-text report of the executed
|
|
tests and show which ones failed:
|
|
.Bd -literal -offset indent
|
|
$ kyua report --verbose -r <.db file from output of test>
|
|
.Ed
|
|
.Pp
|
|
This example would generate an HTML report ready to be published on a
|
|
web server:
|
|
.Bd -literal -offset indent
|
|
$ kyua report-html --output ~/public_html/tests
|
|
.Ed
|
|
.Pp
|
|
For further details on the command-line interface of Kyua, please refer
|
|
to its manual page
|
|
.Xr kyua 1 .
|
|
.Ss Configuring the tests
|
|
Some test cases in the
|
|
.Fx
|
|
Test Suite require manual configuration by the administrator before they can be
|
|
run.
|
|
Unless certain properties are defined, the tests that require them will be
|
|
skipped.
|
|
.Pp
|
|
Test suites are configured by defining their configuration
|
|
variables in
|
|
.Pa /etc/kyua/kyua.conf .
|
|
The format of this file is detailed in
|
|
.Xr kyua.conf 5 .
|
|
.Pp
|
|
The following configuration variables are available in the
|
|
.Fx
|
|
Test Suite:
|
|
.Bl -tag -width "allow_sysctl_side_effects"
|
|
.It allow_devfs_side_effects
|
|
If defined, enables tests that may destroy and recreate semipermanent device
|
|
nodes, like disk devices.
|
|
Without this variable, tests may still create and destroy devices nodes that
|
|
are normally transient, like /dev/tap* and /dev/pts*, as long as they clean
|
|
them up afterwards.
|
|
However, tests that require this variable have a relaxed cleanup requirement;
|
|
they must recreate any devices that they destroyed, but not necessarily with
|
|
the same devnames.
|
|
.It allow_sysctl_side_effects
|
|
Enables tests that change globally significant
|
|
.Xr sysctl 8
|
|
variables.
|
|
The tests will undo any changes in their cleanup phases.
|
|
.It disks
|
|
Must be set to a space delimited list of disk device nodes.
|
|
Tests that need destructive access to disks must use these devices.
|
|
Tests are not required to preserve any data present on these disks.
|
|
.It fibs
|
|
Must be set to a space delimited list of FIBs (routing tables).
|
|
Tests that need to modify a routing table may use any of these.
|
|
Tests will cleanup any new routes that they create.
|
|
.El
|
|
.Ss What to do if something fails?
|
|
If there is
|
|
.Em any failure
|
|
during the execution of the test suite, please consider reporting it to the
|
|
.Fx
|
|
developers so that the failure can be analyzed and fixed.
|
|
To do so, either send a message to the appropriate mailing list or file a
|
|
problem report.
|
|
For more details please refer to:
|
|
.Bl -bullet -offset indent -compact
|
|
.It
|
|
.Lk https://lists.freebsd.org/ "FreeBSD Mailing Lists"
|
|
.It
|
|
.Lk https://www.freebsd.org/support/ "Problem Reporting"
|
|
.El
|
|
.Sh FILES
|
|
.Bl -tag -compact -width "/etc/kyua/kyua.conf"
|
|
.It Pa /etc/kyua/kyua.conf
|
|
System-wide configuration file for
|
|
.Xr kyua 1 .
|
|
.It Pa ~/.kyua/kyua.conf
|
|
User-specific configuration file for
|
|
.Xr kyua 1 ;
|
|
overrides the system file.
|
|
.It Pa ~/.kyua/store.db
|
|
Default result database used by Kyua.
|
|
.It Pa /usr/tests/
|
|
Location of the
|
|
.Fx
|
|
Test Suite.
|
|
.It Pa /usr/tests/Kyuafile
|
|
Top-level test suite definition file.
|
|
.El
|
|
.Sh SEE ALSO
|
|
.Xr kyua 1 ,
|
|
.Xr atf 7 ,
|
|
.Xr build 7 ,
|
|
.Xr development 7
|
|
.Sh HISTORY
|
|
The
|
|
.Fx
|
|
Test Suite first appeared in
|
|
.Fx 10.1
|
|
and was installed by default in
|
|
.Fx 11.0 .
|
|
.Pp
|
|
The
|
|
.Nm
|
|
manual page first appeared in
|
|
.Nx 6.0
|
|
and was later ported to
|
|
.Fx 10.1 .
|
|
.Pp
|
|
The test driver,
|
|
.Xr kyua 1 ,
|
|
was imported as part of the base system in
|
|
.Fx 13.0 ,
|
|
previously being available only in
|
|
.Xr ports 7 .
|
|
.Sh AUTHORS
|
|
.An Julio Merino Aq Mt jmmv@FreeBSD.org
|