d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

share/man/man7/tests.7

Browse Source 
Added a section on test suite configuration, and cleaned up up
	grammar errors and awkward prose.  The config variables were
	discussed on freebsd-testing.

Discussed with: Garrett Cooper, jmmv
MFC after:	2 weeks
Sponsored by:	Spectra Logic Corporation
This commit is contained in:
asomers 2014-03-20 17:13:07 +00:00
parent 8708435e96
commit f0be5aaf96
1 changed files with 62 additions and 49 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

111
share/man/man7/tests.7
View File

@ -26,7 +26,7 @@
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.\"
.Dd October 19, 2013
.Dd March 20, 2014
.Dt TESTS 7
.Os
.Sh NAME
@ -36,30 +36,29 @@
The
.Fx
test suite provides a collection of automated tests for two major purposes.
On the one hand, the test suite aids
On one hand, the test suite aids
.Em developers
in catching bugs and regressions in the code when they performing modifications
to the source tree.
On the other hand, the test suite allows
to detect bugs and regressions when they modify the source tree. On the other
hand, it allows
.Em end users
(and, in particular, system administrators) to verify that fresh installations
of the
.Fx
operating system behave correctly in their hardware platform and also to ensure
that the system does not suffer from regressions during regular system
operation and maintenance.
operating system behave correctly on their hardware platform and also to ensure
that the system does not suffer from regressions during regular operation and
maintenance.
.Pp
The
.Fx
test suite is installed under the
test suite is installed in the
.Pa /usr/tests
hierarchy.
.Pp
This manual page describes how to execute the test suite and how to configure
This manual page describes how to run the test suite and how to configure
some of its optional features.
.Ss When to run the tests?
Before diving into the details of how to run the test suite, here are some
scenarios in which you should be running them:
scenarios in which you should run it:
.Bl -bullet -offset indent
.It
After a fresh installation of
@ -72,8 +71,7 @@ to a different version to ensure that the new code works well on your
hardware platform and that the upgrade did not introduce regressions in your
configuration.
.It
After performing changes to the source tree to catch any bugs and/or regressions
introduced by the modifications.
After modifying the source tree to detect any new bugs and/or regressions.
.It
Periodically, maybe from a
.Xr cron 8
@ -82,18 +80,18 @@ third-party packages or manual modifications to configuration files) do not
introduce unexpected failures.
.El
.Ss Running the tests
First of all, you will need to install the
.Sq ports/devel/kyua
package.
.Pp
Use the following command to run the whole test suite:
First, you will need to install the
.Sq devel/kyua
package from
.Xr ports 7 .
Then use the following command to run the whole test suite:
.Bd -literal -offset indent
$ kyua test -k /usr/tests/Kyuafile
.Ed
.Pp
The above will go through all test programs in
The above will iterate through all test programs in
.Pa /usr/tests
recursively, execute them, store their results and debugging data in Kyua
recursively, execute them, store their results and debugging data in Kyua's
database (by default in
.Pa ~/.kyua/store.db ) ,
and print a summary of the results.
@ -111,15 +109,15 @@ utilities:
$ kyua test -k /usr/tests/Kyuafile bin/cp usr.bin/cut
.Ed
.Ss Obtaining reports of the tests execution
Additional information of the results of the execution can be later extracted
from the database by using the various reporting commands of Kyua.
For example, the following would extract a plain-text report of the executed
Additional information about the test results can be retrieved
by using Kyua's various reporting commands.
For example, the following would print a plain-text report of the executed
tests and show which ones failed:
.Bd -literal -offset indent
$ kyua report
.Ed
.Pp
This other example would generate an HTML report ready to be published on a
This example would generate an HTML report ready to be published on a
web server:
.Bd -literal -offset indent
$ kyua report-html --output ~/public_html/tests
@ -128,30 +126,47 @@ $ kyua report-html --output ~/public_html/tests
For further details on the command-line interface of Kyua, please refer
to its manual page
.Xr kyua 1 .
.\".Ss Configuring the tests
.\"Some test cases in the
.\".Fx
.\"test suite require the administrator to manually set up some configuration
.\"properties before they can run.
.\"Unless these properties are defined, the tests that require them will be marked
.\"as skipped and thus they will not be really executed.
.\".Pp
.\"Test suites are configured by defining the values to their configuration
.\"variables in
.\".Pa /usr/local/etc/kyua/kyua.conf .
.\"The format of this file is detailed in
.\".Xr kyua.conf 5 .
.\".Pp
.\"The following configuration variables are available in the
.\".Fx
.\"test suite:
.\".Bl -tag
.\".It NONE REGISTERED YET
.\"TBD.
.Ss Configuring the tests
Some test cases in the
.Fx
test suite require manual configuration by the administrator before they can be
run. Unless certain properties are defined, the tests that require them will
be skipped.
.Pp
Test suites are configured by defining their configuration
variables in
.Pa /usr/local/etc/kyua/kyua.conf .
The format of this file is detailed in
.Xr kyua.conf 5 .
.Pp
The following configuration variables are available in the
.Fx
test suite:
.Bl -tag -width "allow_sysctl_side_effects"
.It allow_devfs_side_effects
If defined, enables tests that may destroy and recreate semipermanent device
nodes, like disk devices. Without this variable, tests may still create and
destroy devices nodes that are normally transient, like /dev/tap* and
/dev/pts*, as long as they clean them up afterwards. However, tests that
require this variable have a relaxed cleanup requirement; they must recreate
any devices that they destroyed, but not necessarily with the same devnames.
.It allow_sysctl_side_effects
Enables tests that change globally significant
.Xr sysctl 8
variables. The tests will undo any changes in their cleanup phases.
.It disks
Must be set to a space delimited list of disk device nodes. Tests that need
destructive access to disks must use these devices. Tests are not required to
preserve any data present on these disks.
.It fibs
Must be set to a space delimited list of FIBs (routing tables). Tests that
need to modify a routing table may use any of these. Tests will cleanup any
new routes that they create.
.El
.Ss What to do if something fails?
If there is
.Em any failure
during the execution of the test suite, please considering reporting it to the
during the execution of the test suite, please consider reporting it to the
.Fx
developers so that the failure can be analyzed and fixed.
To do so, either send a message to the appropriate mailing list or file a
@ -173,7 +188,7 @@ User-specific configuration file for
.Xr kyua 1 ;
overrides the system file.
.It Pa ~/.kyua/store.db
Default database used by Kyua to maintain the data of the executed tests.
Default result database used by Kyua.
.It Pa /usr/tests/
Location of the
.Fx
@ -184,9 +199,7 @@ Top-level test suite definition file.
.Sh SEE ALSO
.Xr kyua 1 .
.Sh HISTORY
The collection of test programs in
.Pa /usr/tests
first appeared in
This test suite first appeared in
.Fx 11.0 .
.Pp
The