d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Revert r359385-r359387

Browse Source 
As noted by brooks/emaste, this is the wrong approach to take.
Revert the changes so brooks can apply a more proper change.

Requested by:	brooks, emaste
This commit is contained in:
Enji Cooper 2020-03-29 02:40:03 +00:00
parent 9a9f88cc11
commit 09ae09a9eb
15 changed files with 14 additions and 3472 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

95
contrib/kyua/doc/kyua-about.1
View File

@ -1,95 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd May 20, 2015
.Dt KYUA-ABOUT 1
.Os
.Sh NAME
.Nm "kyua about"
.Nd Shows detailed authors, license, and version information
.Sh SYNOPSIS
.Nm
.Op Ar authors | license | version
.Sh DESCRIPTION
The
.Sq about
command provides generic information about the
.Xr kyua 1
tool.
In the default synopsis form (no arguments), the information printed
includes:
.Bl -enum
.It
The name of the package, which is
.Sq kyua .
.It
The version number, which is
.Sq 0.13 .
.It
License information.
.It
Authors information.
.It
A link to the project web site.
.El
.Pp
You can customize the information printed by this command by specifying
the desired topic as the single argument to the command.
This can be one of:
.Bl -tag -width authorsXX
.It Ar authors
Displays the list of authors and contributors only.
.It Ar license
Displays the license information and the list of copyrights.
.It Ar version
Displays the package name and the version number in a format that is
compatible with the output of GNU tools that support a
.Fl -version
flag.
Use this whenever you have to query the version number of the package.
.El
.Sh FILES
The following files are read by the
.Nm
command:
.Bl -tag -width XX
.It Pa /usr/share/doc/kyua/AUTHORS
List of authors (aka copyright holders).
.It Pa /usr/share/doc/kyua/CONTRIBUTORS
List of contributors (aka individuals that have contributed to the project).
.It Pa /usr/share/doc/kyua/LICENSE
License information.
.El
.Sh EXIT STATUS
The
.Nm
command always returns 0.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh SEE ALSO
.Xr kyua 1

59
contrib/kyua/doc/kyua-config.1
View File

@ -1,59 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd September 9, 2012
.Dt KYUA-CONFIG 1
.Os
.Sh NAME
.Nm "kyua config"
.Nd Inspects the values of the loaded configuration
.Sh SYNOPSIS
.Nm
.Op Ar variable1 .. variableN
.Sh DESCRIPTION
The
.Nm
command provides a way to list all defined configuration variables and
their current values.
.Pp
This command is intended to help you in resolving the values of the
configuration variables without having to scan over configuration files.
.Pp
In the default synopsis form (no arguments), the command prints all
configuration variables.
If any arguments are provided, the command will only print the
requested variables.
.Sh EXIT STATUS
The
.Nm
command returns 0 on success or 1 if any of the specified configuration
variables does not exist.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh SEE ALSO
.Xr kyua 1

199
contrib/kyua/doc/kyua-db-exec.1
View File

@ -1,199 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd October 13, 2014
.Dt KYUA-DB-EXEC 1
.Os
.Sh NAME
.Nm "kyua db-exec"
.Nd Executes a SQL statement in a results file
.Sh SYNOPSIS
.Nm
.Op Fl -no-headers
.Op Fl -results-file Ar file
.Ar statement
.Sh DESCRIPTION
The
.Nm
command provides a way to execute an arbitrary SQL statement within the
database.
This command is mostly intended to aid in debugging, but can also be used to
extract information from the database when the current interfaces do not
provide the desired functionality.
.Pp
The input database must exist.
It makes no sense to use
.Nm
on a nonexistent or empty database.
.Pp
The
.Nm
command takes one or more arguments, all of which are concatenated to form
a single SQL statement.
Once the statement is executed,
.Nm
prints the resulting table on the screen, if any.
.Pp
The following subcommand options are recognized:
.Bl -tag -width XX
.It Fl -no-headers
Avoids printing the headers of the table in the output of the command.
.It Fl -results-file Ar path , Fl s Ar path
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Specifies the results file to operate on.
Defaults to
.Sq LATEST ,
which causes
.Nm
to automatically load the latest results file from the current test suite.
.Pp
The following values are accepted:
.Bl -tag -width XX
.It Sq LATEST
Requests the load of the latest results file available for the test suite rooted
at the current directory.
.It Directory
Requests the load of the latest results file available for the test suite rooted
at the given directory.
.It Test suite name
Requests the load of the latest results file available for the given test suite.
.It Results identifier
Requests the load of a specific results file.
.It Explicit file name (aka everything else)
Load the specified results file.
.El
.Pp
See
.Sx Results files
for more details.
.El
.Ss Results files
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Results files contain, as their name implies, the results of the execution of a
test suite.
Each test suite executed by
.Xr kyua-test 1
generates a new results file, and such results files can be loaded later on by
inspection commands such as
.Xr kyua-report 1
to analyze their contents.
.Pp
Results files support identifier-based lookups and also path name lookups.
The differences between the two are described below.
.Pp
The default naming scheme for the results files provides simple support for
identifier-based lookups and historical recording of test suite runs.
Each results file is given an identifier derived from the test suite that
generated it and the time the test suite was run.
Kyua can later look up results files by these fields.
.Pp
The identifier follows this pattern:
.Bd -literal -offset indent
\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt
.Ed
.Pp
where
.Sq test_suite
is the path to the root of the test suite that was run with all slashes replaced
by underscores and
.Sq YYYYMMDD-HHMMSS-uuuuuu
is a timestamp with microsecond resolution.
.Pp
When using the default naming scheme, results files are stored in the
.Pa ~/.kyua/store/
subdirectory and each file holds a name of the form:
.Bd -literal -offset indent
~/.kyua/store/results.\*(Ltidentifier\*(Gt.db
.Ed
.Pp
Results files are simple SQLite databases with the schema described in the
.Pa /usr/share/kyua/store/schema_v?.sql
files.
For details on the schema, please refer to the heavily commented SQL file.
.Sh EXIT STATUS
The
.Nm
command returns 0 on success or 1 if the SQL statement is invalid or fails
to run.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh SEE ALSO
.Xr kyua 1 ,
.Xr kyua-test 1

182
contrib/kyua/doc/kyua-db-migrate.1
View File

@ -1,182 +0,0 @@
.\" Copyright 2013 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd October 13, 2014
.Dt KYUA-DB-MIGRATE 1
.Os
.Sh NAME
.Nm "kyua db-migrate"
.Nd Upgrades the schema of an existing results file
.Sh SYNOPSIS
.Nm
.Op Fl -results-file Ar file
.Sh DESCRIPTION
The
.Nm
command migrates the schema of an existing database to the latest
version implemented in
.Xr kyua 1 .
.Pp
This operation is not reversible.
However, a backup of the database is created in the same directory where the
database lives.
.Pp
The following subcommand options are recognized:
.Bl -tag -width XX
.It Fl -results-file Ar path , Fl s Ar path
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Specifies the results file to operate on.
Defaults to
.Sq LATEST ,
which causes
.Nm
to automatically load the latest results file from the current test suite.
.Pp
The following values are accepted:
.Bl -tag -width XX
.It Sq LATEST
Requests the load of the latest results file available for the test suite rooted
at the current directory.
.It Directory
Requests the load of the latest results file available for the test suite rooted
at the given directory.
.It Test suite name
Requests the load of the latest results file available for the given test suite.
.It Results identifier
Requests the load of a specific results file.
.It Explicit file name (aka everything else)
Load the specified results file.
.El
.Pp
See
.Sx Results files
for more details.
.El
.Ss Results files
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Results files contain, as their name implies, the results of the execution of a
test suite.
Each test suite executed by
.Xr kyua-test 1
generates a new results file, and such results files can be loaded later on by
inspection commands such as
.Xr kyua-report 1
to analyze their contents.
.Pp
Results files support identifier-based lookups and also path name lookups.
The differences between the two are described below.
.Pp
The default naming scheme for the results files provides simple support for
identifier-based lookups and historical recording of test suite runs.
Each results file is given an identifier derived from the test suite that
generated it and the time the test suite was run.
Kyua can later look up results files by these fields.
.Pp
The identifier follows this pattern:
.Bd -literal -offset indent
\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt
.Ed
.Pp
where
.Sq test_suite
is the path to the root of the test suite that was run with all slashes replaced
by underscores and
.Sq YYYYMMDD-HHMMSS-uuuuuu
is a timestamp with microsecond resolution.
.Pp
When using the default naming scheme, results files are stored in the
.Pa ~/.kyua/store/
subdirectory and each file holds a name of the form:
.Bd -literal -offset indent
~/.kyua/store/results.\*(Ltidentifier\*(Gt.db
.Ed
.Pp
Results files are simple SQLite databases with the schema described in the
.Pa /usr/share/kyua/store/schema_v?.sql
files.
For details on the schema, please refer to the heavily commented SQL file.
.Sh EXIT STATUS
The
.Nm
command returns 0 on success or 1 if the migration fails.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh SEE ALSO
.Xr kyua 1

398
contrib/kyua/doc/kyua-debug.1
View File

@ -1,398 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd October 13, 2014
.Dt KYUA-DEBUG 1
.Os
.Sh NAME
.Nm "kyua debug"
.Nd Executes a single test case with facilities for debugging
.Sh SYNOPSIS
.Nm
.Op Fl -build-root Ar path
.Op Fl -kyuafile Ar file
.Op Fl -stdout Ar path
.Op Fl -stderr Ar path
.Ar test_case
.Sh DESCRIPTION
The
.Nm
command provides a mechanism to execute a single test case bypassing some
of the Kyua infrastructure and allowing the user to poke into the execution
behavior of the test.
.Pp
The test case to run is selected by providing a test filter, described below in
.Sx Test filters ,
that matches a single test case.
The test case is executed and its result is printed as the last line of the
output of the tool.
.Pp
The test executed by
.Nm
is run under a controlled environment as described in
.Sx Test isolation .
.Pp
At the moment, the
.Nm
command allows the following aspects of a test case execution to be
tweaked:
.Bl -bullet
.It
Redirection of the test case's stdout and stderr to the console (the
default) or to arbitrary files.
See the
.Fl -stdout
and
.Fl -stderr
options below.
.El
.Pp
The following subcommand options are recognized:
.Bl -tag -width XX
.It Fl -build-root Ar path
Specifies the build root in which to find the test programs referenced
by the Kyuafile, if different from the Kyuafile's directory.
See
.Sx Build directories
below for more information.
.It Fl -kyuafile Ar file , Fl k Ar file
Specifies the Kyuafile to process.
Defaults to
.Pa Kyuafile
file in the current directory.
.It Fl -stderr Ar path
Specifies the file to which to send the standard error of the test
program's body.
The default is
.Pa /dev/stderr ,
which is a special character device that redirects the output to
standard error on the console.
.It Fl -stdout Ar path
Specifies the file to which to send the standard output of the test
program's body.
The default is
.Pa /dev/stdout ,
which is a special character device that redirects the output to
standard output on the console.
.El
.Pp
For example, consider the following Kyua session:
.Bd -literal -offset indent
$ kyua test
kernel/fs:mkdir -> passed
kernel/fs:rmdir -> failed: Invalid argument
1/2 passed (1 failed)
.Ed
.Pp
At this point, we do not have a lot of information regarding the
failure of the
.Sq kernel/fs:rmdir
test.
We can run this test through the
.Nm
command to inspect its output a bit closer, hoping that the test case is
kind enough to log its progress:
.Bd -literal -offset indent
$ kyua debug kernel/fs:rmdir
Trying rmdir('foo')
Trying rmdir(NULL)
kernel/fs:rmdir -> failed: Invalid argument
.Ed
.Pp
Luckily, the offending test case was printing status lines as it
progressed, so we could see the last attempted call and we can know match
the failure message to the problem.
.Ss Build directories
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Em Build directories
(or object directories, target directories, product directories, etc.) is
the concept that allows a developer to keep the source tree clean from
build products by asking the build system to place such build products
under a separate subtree.
.Pp
Most build systems today support build directories.
For example, the GNU Automake/Autoconf build system exposes such concept when
invoked as follows:
.Bd -literal -offset indent
$ cd my-project-1.0
$ mkdir build
$ cd build
$ ../configure
$ make
.Ed
.Pp
Under such invocation, all the results of the build are left in the
.Pa my-project-1.0/build/
subdirectory while maintaining the contents of
.Pa my-project-1.0/
intact.
.Pp
Because build directories are an integral part of most build systems, and
because they are a tool that developers use frequently,
.Nm
supports build directories too.
This manifests in the form of
.Nm
being able to run tests from build directories while reading the (often
immutable) test suite definition from the source tree.
.Pp
One important property of build directories is that they follow (or need to
follow) the exact same layout as the source tree.
For example, consider the following directory listings:
.Bd -literal -offset indent
src/Kyuafile
src/bin/ls/
src/bin/ls/Kyuafile
src/bin/ls/ls.c
src/bin/ls/ls_test.c
src/sbin/su/
src/sbin/su/Kyuafile
src/sbin/su/su.c
src/sbin/su/su_test.c
obj/bin/ls/
obj/bin/ls/ls*
obj/bin/ls/ls_test*
obj/sbin/su/
obj/sbin/su/su*
obj/sbin/su/su_test*
.Ed
.Pp
Note how the directory layout within
.Pa src/
matches that of
.Pa obj/ .
The
.Pa src/
directory contains only source files and the definition of the test suite
(the Kyuafiles), while the
.Pa obj/
directory contains only the binaries generated during a build.
.Pp
All commands that deal with the workspace support the
.Fl -build-root Ar path
option.
When this option is provided, the directory specified by the
option is considered to be the root of the build directory.
For example, considering our previous fake tree layout, we could invoke
.Nm
as any of the following:
.Bd -literal -offset indent
$ kyua debug --kyuafile=src/Kyuafile --build-root=obj
$ cd src && kyua debug --build-root=../obj
.Ed
.Ss Test filters
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
A
.Em test filter
is a string that is used to match test cases or test programs in a test suite.
Filters have the following form:
.Bd -literal -offset indent
test_program_name[:test_case_name]
.Ed
.Pp
Where
.Sq test_program_name
is the name of a test program or a subdirectory in the test suite, and
.Sq test_case_name
is the name of a test case.
.Ss Test isolation
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The test programs and test cases run by
.Nm
are all executed in a deterministic environment.
This known, clean environment serves to make the test execution as
reproducible as possible and also to prevent clashes between tests that may,
for example, create auxiliary files with overlapping names.
.Pp
For plain test programs and for TAP test programs, the whole test program
is run under a single instance of the environment described in this page.
For ATF test programs (see
.Xr atf 7 ) ,
each individual test case
.Em and
test cleanup routine are executed in separate environments.
.Bl -tag -width XX
.It Process space
Each test is executed in an independent processes.
Corollary: the test can do whatever it wants to the current process (such
as modify global variables) without having to undo such changes.
.It Session and process group
The test is executed in its own session and its own process group.
There is no controlling terminal attached to the session.
.Pp
Should the test spawn any children, the children should maintain the same
session and process group.
Modifying any of these settings prevents
.Nm
from being able to kill any stray subprocess as part of the cleanup phase.
If modifying these settings is necessary, or if any subprocess started by
the test decides to use a different process group or session, it is the
responsibility of the test to ensure those subprocesses are forcibly
terminated during cleanup.
.It Work directory
The test is executed in a temporary directory automatically created by the
runtime engine.
Corollary: the test can write to its current directory
without needing to clean any files and/or directories it creates.
The runtime engine takes care to recursively delete the temporary directories
after the execution of a test case.
Any file systems mounted within the temporary directory are also unmounted.
.It Home directory
The
.Va HOME
environment variable is set to the absolute path of the work directory.
.It Umask
The value of the umask is set to 0022.
.It Environment
The
.Va LANG ,
.Va LC_ALL ,
.Va LC_COLLATE ,
.Va LC_CTYPE ,
.Va LC_MESSAGES ,
.Va LC_MONETARY ,
.Va LC_NUMERIC
and
.Va LC_TIME
variables are unset.
.Pp
The
.Va TZ
variable is set to
.Sq UTC .
.Pp
The
.Va TMPDIR
variable is set to the absolute path of the work directory.
This is to prevent the test from mistakenly using a temporary directory
outside of the automatically-managed work directory, should the test use the
.Xr mktemp 3
familiy of functions.
.It Process limits
The maximum soft core size limit is raised to its corresponding hard limit.
This is a simple, best-effort attempt at allowing tests to dump core for
further diagnostic purposes.
.It Configuration varibles
The test engine may pass run-time configuration variables to the test program
via the environment.
The name of the configuration variable is prefixed with
.Sq TEST_ENV_
so that a configuration variable of the form
.Sq foo=bar
becomes accessible in the environment as
.Sq TEST_ENV_foo=bar .
.El
.Sh EXIT STATUS
The
.Nm
command returns 0 if the test case passes or 1 if the test case fails.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh SEE ALSO
.Xr kyua 1 ,
.Xr kyuafile 5

64
contrib/kyua/doc/kyua-help.1
View File

@ -1,64 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd September 9, 2012
.Dt KYUA-HELP 1
.Os
.Sh NAME
.Nm "kyua help"
.Nd Shows usage information
.Sh SYNOPSIS
.Nm
.Op Ar command
.Sh DESCRIPTION
The
.Nm
command provides interactive help on all supported commands and options.
If, for some reason, you happen to spot a discrepancy in the output of this
command and this document, the command is the authoritative source of
information.
.Pp
If no arguments are provided, the command prints the list of common options
and the list of supported subcommands.
.Pp
If the
.Ar command
argument is provided to, this single argument is the name of a valid
subcommand.
In that case,
.Nm
prints a textual description of the command, the list of common options and
the list of subcommand-specific options.
.Sh EXIT STATUS
The
.Nm
command always returns 0.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh SEE ALSO
.Xr kyua 1

232
contrib/kyua/doc/kyua-list.1
View File

@ -1,232 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd October 13, 2014
.Dt KYUA-LIST 1
.Os
.Sh NAME
.Nm "kyua list"
.Nd Lists test cases and their metadata
.Sh SYNOPSIS
.Nm
.Op Fl -build-root Ar path
.Op Fl -kyuafile Ar file
.Op Fl -verbose
.Ar test_case1 Op Ar .. test_caseN
.Sh DESCRIPTION
The
.Nm
command scans all the test programs and test cases in a test suite (as
defined by a
.Xr kyuafile 5 )
and prints a list of all their names, optionally accompanied by any metadata
properties they have.
.Pp
The optional arguments to
.Nm
are used to select which test programs or test cases to run.
These are filters and are described below in
.Sx Test filters .
.Pp
This command must be run within a test suite or a test suite must be
provided with the
.Fl -kyuafile
flag.
.Pp
The following subcommand options are recognized:
.Bl -tag -width XX
.It Fl -build-root Ar path
Specifies the build root in which to find the test programs referenced
by the Kyuafile, if different from the Kyuafile's directory.
See
.Sx Build directories
below for more information.
.It Fl -kyuafile Ar path , Fl k Ar path
Specifies the Kyuafile to process.
Defaults to a
.Pa Kyuafile
file in the current directory.
.It Fl -verbose , Fl v
Prints metadata properties for every test case.
.El
.Ss Build directories
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Em Build directories
(or object directories, target directories, product directories, etc.) is
the concept that allows a developer to keep the source tree clean from
build products by asking the build system to place such build products
under a separate subtree.
.Pp
Most build systems today support build directories.
For example, the GNU Automake/Autoconf build system exposes such concept when
invoked as follows:
.Bd -literal -offset indent
$ cd my-project-1.0
$ mkdir build
$ cd build
$ ../configure
$ make
.Ed
.Pp
Under such invocation, all the results of the build are left in the
.Pa my-project-1.0/build/
subdirectory while maintaining the contents of
.Pa my-project-1.0/
intact.
.Pp
Because build directories are an integral part of most build systems, and
because they are a tool that developers use frequently,
.Nm
supports build directories too.
This manifests in the form of
.Nm
being able to run tests from build directories while reading the (often
immutable) test suite definition from the source tree.
.Pp
One important property of build directories is that they follow (or need to
follow) the exact same layout as the source tree.
For example, consider the following directory listings:
.Bd -literal -offset indent
src/Kyuafile
src/bin/ls/
src/bin/ls/Kyuafile
src/bin/ls/ls.c
src/bin/ls/ls_test.c
src/sbin/su/
src/sbin/su/Kyuafile
src/sbin/su/su.c
src/sbin/su/su_test.c
obj/bin/ls/
obj/bin/ls/ls*
obj/bin/ls/ls_test*
obj/sbin/su/
obj/sbin/su/su*
obj/sbin/su/su_test*
.Ed
.Pp
Note how the directory layout within
.Pa src/
matches that of
.Pa obj/ .
The
.Pa src/
directory contains only source files and the definition of the test suite
(the Kyuafiles), while the
.Pa obj/
directory contains only the binaries generated during a build.
.Pp
All commands that deal with the workspace support the
.Fl -build-root Ar path
option.
When this option is provided, the directory specified by the
option is considered to be the root of the build directory.
For example, considering our previous fake tree layout, we could invoke
.Nm
as any of the following:
.Bd -literal -offset indent
$ kyua list --kyuafile=src/Kyuafile --build-root=obj
$ cd src && kyua list --build-root=../obj
.Ed
.Ss Test filters
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
A
.Em test filter
is a string that is used to match test cases or test programs in a test suite.
Filters have the following form:
.Bd -literal -offset indent
test_program_name[:test_case_name]
.Ed
.Pp
Where
.Sq test_program_name
is the name of a test program or a subdirectory in the test suite, and
.Sq test_case_name
is the name of a test case.
.Sh EXIT STATUS
The
.Nm
command returns 0 on success or 1 if any of the given test case filters
does not match any test case.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh SEE ALSO
.Xr kyua 1 ,
.Xr kyuafile 5

253
contrib/kyua/doc/kyua-report-html.1
View File

@ -1,253 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd October 13, 2014
.Dt KYUA-REPORT-HTML 1
.Os
.Sh NAME
.Nm "kyua report-html"
.Nd Generates an HTML report with the results of a test suite run
.Sh SYNOPSIS
.Nm
.Op Fl -force
.Op Fl -output Ar path
.Op Fl -results-file Ar file
.Op Fl -results-filter Ar types
.Sh DESCRIPTION
The
.Nm
command provides a simple mechanism to generate HTML reports of the
execution of a test suite.
The command processes a results file and then populates a directory with
multiple HTML and supporting files to describe the results recorded in that
results file.
.Pp
The HTML output is static and self-contained, so it can easily be served by
any simple web server.
The command expects the target directory to not exist, because it would
overwrite any contents if not careful.
.Pp
The following subcommand options are recognized:
.Bl -tag -width XX
.It Fl -force
Forces the deletion of the output directory if it exists.
Use care, as this effectively means a
.Sq rm -rf .
.It Fl -output Ar directory
Specifies the target directory into which to generate the HTML files.
The directory must not exist unless the
.Fl -force
option is provided.
The default is
.Pa ./html .
.It Fl -results-file Ar path , Fl s Ar path
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Specifies the results file to operate on.
Defaults to
.Sq LATEST ,
which causes
.Nm
to automatically load the latest results file from the current test suite.
.Pp
The following values are accepted:
.Bl -tag -width XX
.It Sq LATEST
Requests the load of the latest results file available for the test suite rooted
at the current directory.
.It Directory
Requests the load of the latest results file available for the test suite rooted
at the given directory.
.It Test suite name
Requests the load of the latest results file available for the given test suite.
.It Results identifier
Requests the load of a specific results file.
.It Explicit file name (aka everything else)
Load the specified results file.
.El
.Pp
See
.Sx Results files
for more details.
.It Fl -results-filter Ar types
Comma-separated list of the test result types to include in the report.
The ordering of the values is respected so that you can determine how you
want the list of tests to be shown.
.Pp
The valid values are:
.Sq broken ,
.Sq failed ,
.Sq passed ,
.Sq skipped
and
.Sq xfail .
If the parameter supplied to the option is empty, filtering is suppressed
and all result types are shown in the report.
.Pp
The default value for this flag includes all the test results except the
passed tests.
Showing the passed tests by default clutters the report with too much
information, so only abnormal conditions are included.
.El
.Ss Results files
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Results files contain, as their name implies, the results of the execution of a
test suite.
Each test suite executed by
.Xr kyua-test 1
generates a new results file, and such results files can be loaded later on by
inspection commands such as
.Xr kyua-report 1
to analyze their contents.
.Pp
Results files support identifier-based lookups and also path name lookups.
The differences between the two are described below.
.Pp
The default naming scheme for the results files provides simple support for
identifier-based lookups and historical recording of test suite runs.
Each results file is given an identifier derived from the test suite that
generated it and the time the test suite was run.
Kyua can later look up results files by these fields.
.Pp
The identifier follows this pattern:
.Bd -literal -offset indent
\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt
.Ed
.Pp
where
.Sq test_suite
is the path to the root of the test suite that was run with all slashes replaced
by underscores and
.Sq YYYYMMDD-HHMMSS-uuuuuu
is a timestamp with microsecond resolution.
.Pp
When using the default naming scheme, results files are stored in the
.Pa ~/.kyua/store/
subdirectory and each file holds a name of the form:
.Bd -literal -offset indent
~/.kyua/store/results.\*(Ltidentifier\*(Gt.db
.Ed
.Pp
Results files are simple SQLite databases with the schema described in the
.Pa /usr/share/kyua/store/schema_v?.sql
files.
For details on the schema, please refer to the heavily commented SQL file.
.Sh EXIT STATUS
The
.Nm
command always returns 0.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh EXAMPLES
.Ss Workflow with results files
If one runs the following command twice in a row:
.Bd -literal -offset indent
kyua test -k /usr/tests/Kyuafile
.Ed
.Pp
the two executions will generate two different files with names like:
.Bd -literal -offset indent
~/.kyua/store/results.usr_tests.20140731-150500-196784.db
~/.kyua/store/results.usr_tests.20140731-151730-997451.db
.Ed
.Pp
Taking advantage of the default naming scheme, the following commands would all
generate a report for the results of the
.Em latest
execution of the test suite:
.Bd -literal -offset indent
cd /usr/tests && kyua report-html
cd /usr/tests && kyua report-html --results-file=LATEST
kyua report-html --results-file=/usr/tests
kyua report-html --results-file=usr_tests
kyua report-html --results-file=usr_tests.20140731-151730-997451
.Ed
.Pp
But it is also possible to explicitly load data for older runs or from
explicitly-named files:
.Bd -literal -offset indent
kyua report-html \\
--results-file=usr_tests.20140731-150500-196784
kyua report-html \\
--results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db
.Ed
.Sh SEE ALSO
.Xr kyua 1 ,
.Xr kyua-report 1 ,
.Xr kyua-report-junit 1

237
contrib/kyua/doc/kyua-report-junit.1
View File

@ -1,237 +0,0 @@
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd October 13, 2014
.Dt KYUA-REPORT-JUNIT 1
.Os
.Sh NAME
.Nm "kyua report-junit"
.Nd Generates a JUnit report with the results of a test suite run
.Sh SYNOPSIS
.Nm
.Op Fl -output Ar path
.Op Fl -results-file Ar file
.Sh DESCRIPTION
The
.Nm
command provides a simple mechanism to generate JUnit reports of the
execution of a test suite.
The command processes a results file and then generates a single XML file
that complies with the JUnit XSchema.
.Pp
The JUnit output is static and self-contained, so it can easily be plugged
into any continuous integration system, like Jenkins.
.Pp
The following subcommand options are recognized:
.Bl -tag -width XX
.It Fl -output Ar directory
Specifies the file into which to store the JUnit report.
.It Fl -results-file Ar path , Fl s Ar path
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Specifies the results file to operate on.
Defaults to
.Sq LATEST ,
which causes
.Nm
to automatically load the latest results file from the current test suite.
.Pp
The following values are accepted:
.Bl -tag -width XX
.It Sq LATEST
Requests the load of the latest results file available for the test suite rooted
at the current directory.
.It Directory
Requests the load of the latest results file available for the test suite rooted
at the given directory.
.It Test suite name
Requests the load of the latest results file available for the given test suite.
.It Results identifier
Requests the load of a specific results file.
.It Explicit file name (aka everything else)
Load the specified results file.
.El
.Pp
See
.Sx Results files
for more details.
.El
.Ss Caveats
Because of limitations in the JUnit XML schema, not all the data collected by
Kyua can be properly represented in JUnit reports.
However, because test data are extremely useful for debugging purposes, the
.Nm
command shovels these data into the JUnit output.
In particular:
.Bl -bullet
.It
The test case metadata values are prepended to the test case's standard error
output.
.It
Test cases that report expected failures as their results are recorded as
passed.
The fact that they failed as expected is recorded in the test case's standard
error output along with the corresponding reason.
.El
.Ss Results files
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Results files contain, as their name implies, the results of the execution of a
test suite.
Each test suite executed by
.Xr kyua-test 1
generates a new results file, and such results files can be loaded later on by
inspection commands such as
.Xr kyua-report 1
to analyze their contents.
.Pp
Results files support identifier-based lookups and also path name lookups.
The differences between the two are described below.
.Pp
The default naming scheme for the results files provides simple support for
identifier-based lookups and historical recording of test suite runs.
Each results file is given an identifier derived from the test suite that
generated it and the time the test suite was run.
Kyua can later look up results files by these fields.
.Pp
The identifier follows this pattern:
.Bd -literal -offset indent
\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt
.Ed
.Pp
where
.Sq test_suite
is the path to the root of the test suite that was run with all slashes replaced
by underscores and
.Sq YYYYMMDD-HHMMSS-uuuuuu
is a timestamp with microsecond resolution.
.Pp
When using the default naming scheme, results files are stored in the
.Pa ~/.kyua/store/
subdirectory and each file holds a name of the form:
.Bd -literal -offset indent
~/.kyua/store/results.\*(Ltidentifier\*(Gt.db
.Ed
.Pp
Results files are simple SQLite databases with the schema described in the
.Pa /usr/share/kyua/store/schema_v?.sql
files.
For details on the schema, please refer to the heavily commented SQL file.
.Sh EXIT STATUS
The
.Nm
command always returns 0.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh EXAMPLES
.Ss Workflow with results files
If one runs the following command twice in a row:
.Bd -literal -offset indent
kyua test -k /usr/tests/Kyuafile
.Ed
.Pp
the two executions will generate two different files with names like:
.Bd -literal -offset indent
~/.kyua/store/results.usr_tests.20140731-150500-196784.db
~/.kyua/store/results.usr_tests.20140731-151730-997451.db
.Ed
.Pp
Taking advantage of the default naming scheme, the following commands would all
generate a report for the results of the
.Em latest
execution of the test suite:
.Bd -literal -offset indent
cd /usr/tests && kyua report-junit
cd /usr/tests && kyua report-junit --results-file=LATEST
kyua report-junit --results-file=/usr/tests
kyua report-junit --results-file=usr_tests
kyua report-junit --results-file=usr_tests.20140731-151730-997451
.Ed
.Pp
But it is also possible to explicitly load data for older runs or from
explicitly-named files:
.Bd -literal -offset indent
kyua report-junit \\
--results-file=usr_tests.20140731-150500-196784
kyua report-junit \\
--results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db
.Ed
.Sh SEE ALSO
.Xr kyua 1 ,
.Xr kyua-report 1 ,
.Xr kyua-report-html 1

307
contrib/kyua/doc/kyua-report.1
View File

@ -1,307 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd October 13, 2014
.Dt KYUA-REPORT 1
.Os
.Sh NAME
.Nm "kyua report"
.Nd Generates reports with the results of a test suite run
.Sh SYNOPSIS
.Nm
.Op Fl -output Ar path
.Op Fl -results-file Ar file
.Op Fl -results-filter Ar types
.Op Fl -verbose
.Op Ar test_filter1 .. test_filterN
.Sh DESCRIPTION
The
.Nm
command parses a results file and generates a user-friendly, plaintext
report for user consumption on the terminal.
By default, these reports only display a summary of the execution of the full
test suite to highlight where problems may lie.
.Pp
The output of
.Nm
can be customized to display full details on all executed test cases.
Additionally, the optional arguments to
.Nm
are used to select which test programs or test cases to display.
These are filters and are described below in
.Sx Test filters .
.Pp
Reports generated by
.Nm
are
.Em not intended to be machine-parseable .
.Pp
The following subcommand options are recognized:
.Bl -tag -width XX
.It Fl -output Ar path
Specifies the path to which the report should be written to.
The special values
.Pa /dev/stdout
and
.Pa /dev/stderr
can be used to specify the standard output and the standard error,
respectively.
.It Fl -results-file Ar path , Fl s Ar path
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Specifies the results file to operate on.
Defaults to
.Sq LATEST ,
which causes
.Nm
to automatically load the latest results file from the current test suite.
.Pp
The following values are accepted:
.Bl -tag -width XX
.It Sq LATEST
Requests the load of the latest results file available for the test suite rooted
at the current directory.
.It Directory
Requests the load of the latest results file available for the test suite rooted
at the given directory.
.It Test suite name
Requests the load of the latest results file available for the given test suite.
.It Results identifier
Requests the load of a specific results file.
.It Explicit file name (aka everything else)
Load the specified results file.
.El
.Pp
See
.Sx Results files
for more details.
.It Fl -results-filter Ar types
Comma-separated list of the test result types to include in the report.
The ordering of the values is respected so that you can determine how you
want the list of tests to be shown.
.Pp
The valid values are:
.Sq broken ,
.Sq failed ,
.Sq passed ,
.Sq skipped
and
.Sq xfail .
If the parameter supplied to the option is empty, filtering is suppressed
and all result types are shown in the report.
.Pp
The default value for this flag includes all the test results except the
passed tests.
Showing the passed tests by default clutters the report with too much
information, so only abnormal conditions are included.
.It Fl -verbose
Prints a detailed report of the execution.
In addition to all the information printed by default, verbose reports
include the runtime context of the test suite run, the metadata of each
test case, and the verbatim output of the test cases.
.El
.Ss Results files
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Results files contain, as their name implies, the results of the execution of a
test suite.
Each test suite executed by
.Xr kyua-test 1
generates a new results file, and such results files can be loaded later on by
inspection commands such as
.Xr kyua-report 1
to analyze their contents.
.Pp
Results files support identifier-based lookups and also path name lookups.
The differences between the two are described below.
.Pp
The default naming scheme for the results files provides simple support for
identifier-based lookups and historical recording of test suite runs.
Each results file is given an identifier derived from the test suite that
generated it and the time the test suite was run.
Kyua can later look up results files by these fields.
.Pp
The identifier follows this pattern:
.Bd -literal -offset indent
\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt
.Ed
.Pp
where
.Sq test_suite
is the path to the root of the test suite that was run with all slashes replaced
by underscores and
.Sq YYYYMMDD-HHMMSS-uuuuuu
is a timestamp with microsecond resolution.
.Pp
When using the default naming scheme, results files are stored in the
.Pa ~/.kyua/store/
subdirectory and each file holds a name of the form:
.Bd -literal -offset indent
~/.kyua/store/results.\*(Ltidentifier\*(Gt.db
.Ed
.Pp
Results files are simple SQLite databases with the schema described in the
.Pa /usr/share/kyua/store/schema_v?.sql
files.
For details on the schema, please refer to the heavily commented SQL file.
.Ss Test filters
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
A
.Em test filter
is a string that is used to match test cases or test programs in a test suite.
Filters have the following form:
.Bd -literal -offset indent
test_program_name[:test_case_name]
.Ed
.Pp
Where
.Sq test_program_name
is the name of a test program or a subdirectory in the test suite, and
.Sq test_case_name
is the name of a test case.
.Sh EXIT STATUS
The
.Nm
command returns 0 if no filters were specified or if all filters match one
or more test cases.
If any filter fails to match any test case, the command returns 1.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh EXAMPLES
.Ss Workflow with results files
If one runs the following command twice in a row:
.Bd -literal -offset indent
kyua test -k /usr/tests/Kyuafile
.Ed
.Pp
the two executions will generate two different files with names like:
.Bd -literal -offset indent
~/.kyua/store/results.usr_tests.20140731-150500-196784.db
~/.kyua/store/results.usr_tests.20140731-151730-997451.db
.Ed
.Pp
Taking advantage of the default naming scheme, the following commands would all
generate a report for the results of the
.Em latest
execution of the test suite:
.Bd -literal -offset indent
cd /usr/tests && kyua report
cd /usr/tests && kyua report --results-file=LATEST
kyua report --results-file=/usr/tests
kyua report --results-file=usr_tests
kyua report --results-file=usr_tests.20140731-151730-997451
.Ed
.Pp
But it is also possible to explicitly load data for older runs or from
explicitly-named files:
.Bd -literal -offset indent
kyua report \\
--results-file=usr_tests.20140731-150500-196784
kyua report \\
--results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db
.Ed
.Sh SEE ALSO
.Xr kyua 1 ,
.Xr kyua-report-html 1 ,
.Xr kyua-report-junit 1

498
contrib/kyua/doc/kyua-test.1
View File

@ -1,498 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd October 13, 2014
.Dt KYUA-TEST 1
.Os
.Sh NAME
.Nm "kyua test"
.Nd Runs tests
.Sh SYNOPSIS
.Nm
.Op Fl -build-root Ar path
.Op Fl -kyuafile Ar file
.Op Fl -results-file Ar file
.Op Ar test_filter1 .. test_filterN
.Sh DESCRIPTION
The
.Nm
command loads a test suite definition from a
.Xr kyuafile 5 ,
runs the tests defined in it, and records the results into a new results
file.
By default, all tests in the test suite are executed but the optional
arguments to
.Nm
can be used to select which test programs or test cases to run.
These are filters and are described below in
.Sx Test filters .
.Pp
Every test executed by
.Nm
is run under a controlled environment as described in
.Sx Test isolation .
.Pp
The following subcommand options are recognized:
.Bl -tag -width XX
.It Fl -build-root Ar path
Specifies the build root in which to find the test programs referenced by
the Kyuafile, if different from the Kyuafile's directory.
See
.Sx Build directories
below for more information.
.It Fl -kyuafile Ar path , Fl k Ar path
Specifies the Kyuafile to process.
Defaults to a
.Pa Kyuafile
file in the current directory.
.It Fl -results-file Ar path , Fl s Ar path
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Specifies the results file to create.
Defaults to
.Sq LATEST ,
which causes
.Nm
to automatically generate a new results file for the test run.
.Pp
The following values are accepted:
.Bl -tag -width XX
.It Sq NEW
Requests the automatic generation of a new results filename based on the test
suite being run and the current time.
.It Explicit filename (aka everything else)
Store the results file where indicated.
.El
.Pp
See
.Sx Results files
for more details.
.El
.Pp
You can later inspect the results of the test run in more detail by using
.Xr kyua-report 1
or you can execute a single test case with debugging functionality by using
.Xr kyua-debug 1 .
.Ss Build directories
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Em Build directories
(or object directories, target directories, product directories, etc.) is
the concept that allows a developer to keep the source tree clean from
build products by asking the build system to place such build products
under a separate subtree.
.Pp
Most build systems today support build directories.
For example, the GNU Automake/Autoconf build system exposes such concept when
invoked as follows:
.Bd -literal -offset indent
$ cd my-project-1.0
$ mkdir build
$ cd build
$ ../configure
$ make
.Ed
.Pp
Under such invocation, all the results of the build are left in the
.Pa my-project-1.0/build/
subdirectory while maintaining the contents of
.Pa my-project-1.0/
intact.
.Pp
Because build directories are an integral part of most build systems, and
because they are a tool that developers use frequently,
.Nm
supports build directories too.
This manifests in the form of
.Nm
being able to run tests from build directories while reading the (often
immutable) test suite definition from the source tree.
.Pp
One important property of build directories is that they follow (or need to
follow) the exact same layout as the source tree.
For example, consider the following directory listings:
.Bd -literal -offset indent
src/Kyuafile
src/bin/ls/
src/bin/ls/Kyuafile
src/bin/ls/ls.c
src/bin/ls/ls_test.c
src/sbin/su/
src/sbin/su/Kyuafile
src/sbin/su/su.c
src/sbin/su/su_test.c
obj/bin/ls/
obj/bin/ls/ls*
obj/bin/ls/ls_test*
obj/sbin/su/
obj/sbin/su/su*
obj/sbin/su/su_test*
.Ed
.Pp
Note how the directory layout within
.Pa src/
matches that of
.Pa obj/ .
The
.Pa src/
directory contains only source files and the definition of the test suite
(the Kyuafiles), while the
.Pa obj/
directory contains only the binaries generated during a build.
.Pp
All commands that deal with the workspace support the
.Fl -build-root Ar path
option.
When this option is provided, the directory specified by the
option is considered to be the root of the build directory.
For example, considering our previous fake tree layout, we could invoke
.Nm
as any of the following:
.Bd -literal -offset indent
$ kyua test --kyuafile=src/Kyuafile --build-root=obj
$ cd src && kyua test --build-root=../obj
.Ed
.Ss Results files
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Results files contain, as their name implies, the results of the execution of a
test suite.
Each test suite executed by
.Xr kyua-test 1
generates a new results file, and such results files can be loaded later on by
inspection commands such as
.Xr kyua-report 1
to analyze their contents.
.Pp
Results files support identifier-based lookups and also path name lookups.
The differences between the two are described below.
.Pp
The default naming scheme for the results files provides simple support for
identifier-based lookups and historical recording of test suite runs.
Each results file is given an identifier derived from the test suite that
generated it and the time the test suite was run.
Kyua can later look up results files by these fields.
.Pp
The identifier follows this pattern:
.Bd -literal -offset indent
\*(Lttest_suite\*(Gt.\*(LtYYYYMMDD\*(Gt-\*(LtHHMMSS\*(Gt-\*(Ltuuuuuu\*(Gt
.Ed
.Pp
where
.Sq test_suite
is the path to the root of the test suite that was run with all slashes replaced
by underscores and
.Sq YYYYMMDD-HHMMSS-uuuuuu
is a timestamp with microsecond resolution.
.Pp
When using the default naming scheme, results files are stored in the
.Pa ~/.kyua/store/
subdirectory and each file holds a name of the form:
.Bd -literal -offset indent
~/.kyua/store/results.\*(Ltidentifier\*(Gt.db
.Ed
.Pp
Results files are simple SQLite databases with the schema described in the
.Pa /usr/share/kyua/store/schema_v?.sql
files.
For details on the schema, please refer to the heavily commented SQL file.
.Ss Test filters
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
A
.Em test filter
is a string that is used to match test cases or test programs in a test suite.
Filters have the following form:
.Bd -literal -offset indent
test_program_name[:test_case_name]
.Ed
.Pp
Where
.Sq test_program_name
is the name of a test program or a subdirectory in the test suite, and
.Sq test_case_name
is the name of a test case.
.Ss Test isolation
.\" Copyright 2014 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The test programs and test cases run by
.Nm
are all executed in a deterministic environment.
This known, clean environment serves to make the test execution as
reproducible as possible and also to prevent clashes between tests that may,
for example, create auxiliary files with overlapping names.
.Pp
For plain test programs and for TAP test programs, the whole test program
is run under a single instance of the environment described in this page.
For ATF test programs (see
.Xr atf 7 ) ,
each individual test case
.Em and
test cleanup routine are executed in separate environments.
.Bl -tag -width XX
.It Process space
Each test is executed in an independent processes.
Corollary: the test can do whatever it wants to the current process (such
as modify global variables) without having to undo such changes.
.It Session and process group
The test is executed in its own session and its own process group.
There is no controlling terminal attached to the session.
.Pp
Should the test spawn any children, the children should maintain the same
session and process group.
Modifying any of these settings prevents
.Nm
from being able to kill any stray subprocess as part of the cleanup phase.
If modifying these settings is necessary, or if any subprocess started by
the test decides to use a different process group or session, it is the
responsibility of the test to ensure those subprocesses are forcibly
terminated during cleanup.
.It Work directory
The test is executed in a temporary directory automatically created by the
runtime engine.
Corollary: the test can write to its current directory
without needing to clean any files and/or directories it creates.
The runtime engine takes care to recursively delete the temporary directories
after the execution of a test case.
Any file systems mounted within the temporary directory are also unmounted.
.It Home directory
The
.Va HOME
environment variable is set to the absolute path of the work directory.
.It Umask
The value of the umask is set to 0022.
.It Environment
The
.Va LANG ,
.Va LC_ALL ,
.Va LC_COLLATE ,
.Va LC_CTYPE ,
.Va LC_MESSAGES ,
.Va LC_MONETARY ,
.Va LC_NUMERIC
and
.Va LC_TIME
variables are unset.
.Pp
The
.Va TZ
variable is set to
.Sq UTC .
.Pp
The
.Va TMPDIR
variable is set to the absolute path of the work directory.
This is to prevent the test from mistakenly using a temporary directory
outside of the automatically-managed work directory, should the test use the
.Xr mktemp 3
familiy of functions.
.It Process limits
The maximum soft core size limit is raised to its corresponding hard limit.
This is a simple, best-effort attempt at allowing tests to dump core for
further diagnostic purposes.
.It Configuration varibles
The test engine may pass run-time configuration variables to the test program
via the environment.
The name of the configuration variable is prefixed with
.Sq TEST_ENV_
so that a configuration variable of the form
.Sq foo=bar
becomes accessible in the environment as
.Sq TEST_ENV_foo=bar .
.El
.Sh EXIT STATUS
The
.Nm
command returns 0 if all executed test cases pass or 1 if any of the
executed test cases fails or if any of the given test case filters does not
match any test case.
.Pp
Additional exit codes may be returned as described in
.Xr kyua 1 .
.Sh EXAMPLES
.Ss Workflow with results files
If one runs the following command twice in a row:
.Bd -literal -offset indent
kyua test -k /usr/tests/Kyuafile
.Ed
.Pp
the two executions will generate two different files with names like:
.Bd -literal -offset indent
~/.kyua/store/results.usr_tests.20140731-150500-196784.db
~/.kyua/store/results.usr_tests.20140731-151730-997451.db
.Ed
.Pp
Taking advantage of the default naming scheme, the following commands would all
generate a report for the results of the
.Em latest
execution of the test suite:
.Bd -literal -offset indent
cd /usr/tests && kyua report
cd /usr/tests && kyua report --results-file=LATEST
kyua report --results-file=/usr/tests
kyua report --results-file=usr_tests
kyua report --results-file=usr_tests.20140731-151730-997451
.Ed
.Pp
But it is also possible to explicitly load data for older runs or from
explicitly-named files:
.Bd -literal -offset indent
kyua report \\
--results-file=usr_tests.20140731-150500-196784
kyua report \\
--results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db
.Ed
.Sh SEE ALSO
.Xr kyua 1 ,
.Xr kyua-report 1 ,
.Xr kyuafile 5

400
contrib/kyua/doc/kyua.1
View File

@ -1,400 +0,0 @@
.\" Copyright 2011 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd May 12, 2015
.Dt KYUA 1
.Os
.Sh NAME
.Nm kyua
.Nd Testing framework for infrastructure software
.Sh SYNOPSIS
.Nm
.Op Fl -config Ar file
.Op Fl -logfile Ar file
.Op Fl -loglevel Ar level
.Op Fl -variable Ar name=value
.Ar command
.Op Ar command_options
.Op Ar command_arguments
.Sh DESCRIPTION
.Em If you are here looking for details on how to run the test suite in
.Pa /usr/tests
.Em ( or
.Pa /usr/tests ) ,
.Em please start by reading the
.Xr tests 7
.Em manual page that should be supplied by your system .
.Pp
Kyua is a testing framework for infrastructure software, originally
designed to equip BSD-based operating systems with a test suite.
This means that Kyua is lightweight and simple, and that Kyua integrates well
with various build systems and continuous integration frameworks.
.Pp
Kyua features an expressive test suite definition language, a safe
runtime engine for test suites and a powerful report generation engine.
.Pp
Kyua is for both developers and users, from the developer applying a
simple fix to a library to the system administrator deploying a new
release on a production machine.
.Pp
Kyua is able to execute test programs written with a plethora of testing
libraries and languages.
The test program library of choice is ATF, which
.Nm Ns 's
design originated from.
However, framework-less test programs and TAP-compliant test programs can also
be executed through
.Nm
.Ss Overview
As can be observed in the synopsis, the interface of
.Nm
implements a common subcommand-based interface.
The arguments to the tool specify, in this order: a set of common options
that all the commands accept, a required
.Ar command
name that specifies what
.Nm
should do, and
a set of possibly-optional
.Ar command_options
and
.Ar command_arguments
that are specific to the chosen command.
.Pp
The following options are recognized by all the commands.
Keep in mind that these must always be specified before the command name.
.Bl -tag -width XX
.It Fl -config Ar path , Fl c Ar path
Specifies the configuration file to process, which must be in the format
described in
.Xr kyua.conf 5 .
The special value
.Sq none
explicitly disables the loading of any configuration file.
.Pp
Defaults to
.Pa ~/.kyua/kyua.conf
if it exists, otherwise to
.Pa /etc/kyua/kyua.conf
if it exists,
or else to
.Sq none .
.It Fl -logfile Ar path
Specifies the location of the file to which
.Nm
will log run time events useful for postmortem debugging.
.Pp
The default depends on different environment variables as described in
.Sx Logging ,
but typically the file will be stored within the user's home directory.
.It Fl -loglevel Ar level
Specifies the maximum logging level to record in the log file.
See
.Sx Logging
for more details.
.Pp
The default is
.Sq info .
.It Fl -variable Ar name=value , Fl v Ar name=value
Sets the
.Ar name
configuration variable to
.Ar value .
The values set through this option have preference over the values set in the
configuration file.
.Pp
The specified variable can either be a builtin variable or a test-suite
specific variable.
See
.Xr kyua.conf 5
for more details.
.El
.Pp
The following commands are generic and do not have any relation to the execution
of tests or the inspection of their results:
.Bl -tag -width reportXjunitXX -offset indent
.It Ar about
Shows general program information.
See
.Xr kyua-about 1 .
.It Ar config
Inspects the values of the configuration variables.
See
.Xr kyua-config 1 .
.It Ar db-exec
Executes an arbitrary SQL statement on a results file and prints the
resulting table.
See
.Xr kyua-db-exec 1 .
.It Ar help
Shows usage information.
See
.Xr kyua-help 1 .
.El
.Pp
The following commands are used to generate reports based on the data previously
recorded in a results file:
.Bl -tag -width reportXjunitXX -offset indent
.It Ar report
Generates a plaintext report.
Combined with its
.Fl -verbose
flag and the ability to only display specific test cases, this command can also
be used to debug test failures post-facto on the console.
See
.Xr kyua-report 1 .
.It Ar report-html
Generates an HTML report.
See
.Xr kyua-report-html 1 .
.It Ar report-junit
Generates a JUnit report.
See
.Xr kyua-report-junit 1 .
.El
.Pp
The following commands are used to interact with a test suite:
.Bl -tag -width reportXjunitXX -offset indent
.It Ar debug
Executes a single test case in a controlled environment for debugging purposes.
See
.Xr kyua-debug 1 .
.It Ar list
Lists test cases defined in a test suite by a
.Xr kyuafile 5
and, optionally, displays their metadata.
See
.Xr kyua-list 1 .
.It Ar test
Runs tests defined in a test suite by a
.Xr kyuafile 5 .
See
.Xr kyua-test 1 .
.El
.Ss Logging
.Nm
has a logging facility that collects all kinds of events at run time.
These events are always logged to a file so that the log is available when
it is most needed: right after a non-reproducible problem happens.
The only way to disable logging is by sending the log to
.Pa /dev/null .
.Pp
The location of the log file can be manually specified with the
.Fl -logfile
option, which applies to all commands.
If no file is explicitly specified, the location of the log files is chosen in
this order:
.Bl -enum -offset indent
.It
.Pa ${HOME}/.kyua/logs/
if
.Va HOME
is defined.
.It
.Pa ${TMPDIR}/
if
.Va TMPDIR
is defined.
.It
.Pa /tmp/ .
.El
.Pp
And the default naming scheme of the log files is:
.Sq <progname>.<timestamp>.log .
.Pp
The messages stored in the log file have a level (or severity) attached to
them.
These are:
.Bl -tag -width warningXX -offset indent
.It error
Fatal error messages.
The program generally terminates after these, either in a clean manner or by
crashing.
.It warning
Non-fatal error messages.
These generally report a condition that must be addressed but the application
can continue to run.
.It info
Informational messages.
These tell the user what the program was doing at a general level of
operation.
.It debug
Detailed informational messages.
These are often useful when debugging problems in the application, as they
contain lots of internal details.
.El
.Pp
The default log level is
.Sq info
unless explicitly overridden with
.Fl -loglevel .
.Pp
The log file is a plain text file containing one line per log record.
The format of each line is as follows:
.Bd -literal -offset indent
timestamp entry_type pid file:line: message
.Ed
.Pp
.Ar entry_type
can be one of:
.Sq E
for an error,
.Sq W
for a warning,
.Sq I
for an informational message and
.Sq D
for a debug message.
.Ss Bug reporting
If you think you have encountered a bug in
.Nm ,
please take the time to let the developers know about it.
This will ensure that the bug is addressed and potentially fixed in the next
Kyua release.
.Pp
The first step in reporting a bug is to check if there already is a similar
bug in the database.
You can check what issues are currently in the database by going to:
.Bd -literal -offset indent
https://github.com/jmmv/kyua/issues/
.Ed
.Pp
If there is no existing issue that describes an issue similar to the
one you are experiencing, you can open a new one by visiting:
.Bd -literal -offset indent
https://github.com/jmmv/kyua/issues/new/
.Ed
.Pp
When doing so, please include as much detail as possible.
Among other things, explain what operating system and platform you are running
.Nm
on, what were you trying to do, what exact messages you saw on the screen,
how did you expect the program to behave, and any other details that you
may find relevant.
.Pp
Also, please include a copy of the log file corresponding to the problem
you are experiencing.
Unless you have changed the location of the log files, you can most likely
find them in
.Pa ~/.kyua/logs/ .
If the problem is reproducible, it is good idea to regenerate the log file
with an increased log level so as to provide more information.
For example:
.Bd -literal -offset indent
$ kyua --logfile=problem.log --loglevel=debug \\
[rest of the command line]
.Ed
.Sh ENVIRONMENT
The following variables are recognized and can be freely tuned by the end user:
.Bl -tag -width COLUMNSXX
.It Va COLUMNS
The width of the screen, in number of characters.
.Nm
uses this to wrap long lines.
If not present, the width of the screen is determined from the terminal
stdout is connected to, and, if the guessing fails, this defaults to infinity.
.It Va HOME
Path to the user's home directory.
.Nm
uses this location to determine paths to configuration files and default log
files.
.It Va TMPDIR
Path to the system-wide temporary directory.
.Nm
uses this location to place the work directory of test cases, among other
things.
.Pp
The default value of this variable depends on the operating system.
In general, it is
.Pa /tmp .
.El
.Pp
The following variables are also recognized, but you should not need to set them
during normal operation.
They are only provided to override the value of built-in values, which is useful
when testing
.Nm
itself:
.Bl -tag -width KYUAXCONFDIRXX
.It Va KYUA_CONFDIR
Path to the system-wide configuration files for
.Nm .
.Pp
Defaults to
.Pa /etc/kyua .
.It Va KYUA_DOCDIR
Path to the location of installed documentation.
.Pp
Defaults to
.Pa /usr/share/doc/kyua .
.It Va KYUA_MISCDIR
Path to the location of the installed miscellaneous scripts and data
files provided by
.Nm .
.Pp
Defaults to
.Pa /usr/share/kyua/misc .
.It Va KYUA_STOREDIR
Path to the location of the installed store support files; e.g., the
directory containing the SQL database schema.
.Pp
Defaults to
.Pa /usr/share/kyua/store .
.El
.Sh FILES
.Bl -tag -width XXXX
.It Pa ~/.kyua/store/
Default location for the results files.
.It Pa ~/.kyua/kyua.conf
User-specific configuration file.
.It Pa ~/.kyua/logs/
Default location for the collected log files.
.It Pa /etc/kyua/kyua.conf
System-wide configuration file.
.El
.Sh EXIT STATUS
.Nm
returns 0 on success, 1 on a controlled error condition in the given
subcommand, 2 on a general unexpected error and 3 on a usage error.
.Pp
The documentation of the subcommands in the corresponding manual pages only
details the difference between a successful exit (0) and the detection of a
controlled error (1).
Even though when those manual pages do not describe any other exit statuses,
codes above 1 can be returned.
.Sh SEE ALSO
.Xr kyua.conf 5 ,
.Xr kyuafile 5 ,
.Xr atf 7 ,
.Xr tests 7
.Sh AUTHORS
For more details on the people that made
.Nm
possible and the license terms, run:
.Bd -literal -offset indent
$ kyua about
.Ed

141
contrib/kyua/doc/kyua.conf.5
View File

@ -1,141 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd February 20, 2015
.Dt KYUA.CONF 5
.Os
.Sh NAME
.Nm kyua.conf
.Nd Configuration file for the kyua tool
.Sh SYNOPSIS
.Fn syntax "int version"
.Pp
Variables:
.Va architecture ,
.Va platform ,
.Va test_suites ,
.Va unprivileged_user .
.Sh DESCRIPTION
The configuration of Kyua is a simple collection of key/value pairs called
configuration variables.
There are configuration variables that have a special meaning to the runtime
engine implemented by
.Xr kyua 1 ,
and there are variables that only have meaning in the context of particular
test suites.
.Pp
Configuration files are Lua scripts.
In their most basic form, their whole purpose is to assign values to
variables, but the user has the freedom to implement any logic he desires
to compute such values.
.Ss File versioning
Every
.Nm
file starts with a call to
.Fn syntax "int version" .
This call determines the specific schema used by the file so that future
backwards-incompatible modifications to the file can be introduced.
.Pp
Any new
.Nm
file should set
.Fa version
to
.Sq 2 .
.Ss Runtime configuration variables
The following variables are internally recognized by
.Xr kyua 1 :
.Bl -tag -width XX -offset indent
.It Va architecture
Name of the system architecture (aka processor type).
.It Va parallelism
Maximum number of test cases to execute concurrently.
.It Va platform
Name of the system platform (aka machine type).
.It Va unprivileged_user
Name or UID of the unprivileged user.
.Pp
If set, the given user must exist in the system and his privileges will be
used to run test cases that need regular privileges when
.Xr kyua 1
is executed as root.
.El
.Ss Test-suite configuration variables
Each test suite is able to recognize arbitrary configuration variables, and
their type and meaning is specific to the test suite.
Because the existence and naming of these variables depends on every test
suite, this manual page cannot detail them; please refer to the documentation
of the test suite you are working with for more details on this topic.
.Pp
Test-suite specific configuration variables are defined inside the
.Va test_suites
dictionary.
The general syntax is:
.Bd -literal -offset indent
test_suites.<test_suite_name>.<variable_name> = <value>
.Ed
.Pp
where
.Va test_suite_name
is the name of the test suite,
.Va variable_name
is the name of the variable to set, and
.Va value
is a value.
The value can be a string, an integer or a boolean.
.Sh FILES
.Bl -tag -width XX
.It /usr/share/examples/kyua/kyua.conf
Sample configuration file.
.El
.Sh EXAMPLES
The following
.Nm
shows a simple configuration file that overrides a bunch of the built-in
.Xr kyua 1
configuration variables:
.Bd -literal -offset indent
syntax(2)
architecture = 'x86_64'
platform = 'amd64'
.Ed
.Pp
The following is a more complex example that introduces the definition of
per-test suite configuration variables:
.Bd -literal -offset indent
syntax(2)
-- Assign built-in variables.
unprivileged_user = '_tests'
-- Assign test-suite variables. All of these must be strings.
test_suites.NetBSD.file_systems = 'ffs ext2fs'
test_suites.X11.graphics_driver = 'vesa'
.Ed
.Sh SEE ALSO
.Xr kyua 1

407
contrib/kyua/doc/kyuafile.5
View File

@ -1,407 +0,0 @@
.\" Copyright 2012 The Kyua Authors.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions are
.\" met:
.\"
.\" * Redistributions of source code must retain the above copyright
.\" notice, this list of conditions and the following disclaimer.
.\" * Redistributions in binary form must reproduce the above copyright
.\" notice, this list of conditions and the following disclaimer in the
.\" documentation and/or other materials provided with the distribution.
.\" * Neither the name of Google Inc. nor the names of its contributors
.\" may be used to endorse or promote products derived from this software
.\" without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
.\" "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
.\" LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
.\" A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
.\" OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
.\" LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
.\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
.\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
.\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
.\" OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.Dd July 3, 2015
.Dt KYUAFILE 5
.Os
.Sh NAME
.Nm Kyuafile
.Nd Test suite description files
.Sh SYNOPSIS
.Fn atf_test_program "string name" "[string metadata]"
.Fn current_kyuafile
.Fn fs.basename "string path"
.Fn fs.dirname "string path"
.Fn fs.exists "string path"
.Fn fs.files "string path"
.Fn fs.is_absolute "string path"
.Fn fs.join "string path" "string path"
.Fn include "string path"
.Fn plain_test_program "string name" "[string metadata]"
.Fn syntax "int version"
.Fn tap_test_program "string name" "[string metadata]"
.Fn test_suite "string name"
.Sh DESCRIPTION
A test suite is a collection of test programs and is represented by a
hierarchical layout of test binaries on the file system.
Any subtree of the file system can represent a test suite, provided that it
includes one or more
.Nm Ns s ,
which are the test suite definition files.
.Pp
A
.Nm
is a Lua script whose purpose is to describe the structure of the test
suite it belongs to.
To do so, the script has access to a collection of special functions provided
by
.Xr kyua 1
as described in
.Sx Helper functions .
.Ss File versioning
Every
.Nm
file starts with a call to
.Fn syntax "int version" .
This call determines the specific schema used by the file so that future
backwards-incompatible modifications to the file can be introduced.
.Pp
Any new
.Nm
file should set
.Fa version
to
.Sq 2 .
.Ss Test suite definition
If the
.Nm
registers any test programs,
the
.Nm
must define the name of the test suite the test programs belong to by using the
.Fn test_suite
function at the very beginning of the file.
.Pp
The test suite name provided in the
.Fn test_suite
call tells
.Xr kyua 1
which set of configuration variables from
.Xr kyua.conf 5
to pass to the test programs at run time.
.Ss Test program registration
A
.Nm
can register test programs by means of a variety of
.Fn *_test_program
functions, all of which take the name of a test program and a set of
optional metadata properties that describe such test program.
.Pp
The test programs to be registered must live in the current directory; in
other words, the various
.Fn *_test_program
calls cannot reference test programs in other directories.
The rationale for this is to force all
.Nm
files to be self-contained, and to simplify their internal representation.
.Pp
.Em ATF test programs
are those that use the
.Xr atf 7
libraries.
They can be registered with the
.Fn atf_test_program
table constructor.
This function takes the
.Fa name
of the test program and a collection of optional metadata settings for all
the test cases in the test program.
Any metadata properties defined by the test cases themselves override the
metadata values defined here.
.Pp
.Em Plain test programs
are those that return 0 on success and non-0 on failure; in general, most test
programs (even those that use fancy unit-testing libraries) behave this way and
thus also qualify as plain test programs.
They can be registered with the
.Fn plain_test_program
table constructor.
This function takes the
.Fa name
of the test program, an optional
.Fa test_suite
name that overrides the global test suite name, and a collection of optional
metadata settings for the test program.
.Pp
.Em TAP test programs
are those that implement the Test Anything Protocol.
They can be registered with the
.Fn tap_test_program
table constructor.
This function takes the
.Fa name
of the test program and a collection of optional metadata settings for the
test program.
.Pp
The following metadata properties can be passed to any test program definition:
.Bl -tag -width XX -offset indent
.It Va allowed_architectures
Whitespace-separated list of machine architecture names allowed by the test.
If empty or not defined, the test is allowed to run on any machine
architecture.
.It Va allowed_platforms
Whitespace-separated list of machine platform names allowed by the test.
If empty or not defined, the test is allowed to run on any machine
platform.
.It Va custom.NAME
Custom variable defined by the test where
.Sq NAME
denotes the name of the variable.
These variables are useful to tag your tests with information specific to
your project.
The values of such variables are propagated all the way from the tests to the
results files and later to any generated reports.
.Pp
Note that if the name happens to have dashes or any other special characters
in it, you will have to use a special Lua syntax to define the property.
Refer to the
.Sx EXAMPLES
section below for clarification.
.It Va description
Textual description of the test.
.It Va is_exclusive
If true, indicates that this test program cannot be executed along any other
programs at the same time.
Test programs that affect global system state, such as those that modify the
value of a
.Xr sysctl 8
setting, must set themselves as exclusive to prevent failures due to race
conditions.
Defaults to false.
.It Va required_configs
Whitespace-separated list of configuration variables that the test requires
to be defined before it can run.
.It Va required_disk_space
Amount of available disk space that the test needs to run successfully.
.It Va required_files
Whitespace-separated list of paths that the test requires to exist before
it can run.
.It Va required_memory
Amount of physical memory that the test needs to run successfully.
.It Va required_programs
Whitespace-separated list of basenames or absolute paths pointing to executable
binaries that the test requires to exist before it can run.
.It Va required_user
If empty, the test has no restrictions on the calling user for it to run.
If set to
.Sq unprivileged ,
the test needs to not run as root.
If set to
.Sq root ,
the test must run as root.
.It Va timeout
Amount of seconds that the test is allowed to execute before being killed.
.El
.Ss Recursion
To reference test programs in another subdirectory, a different
.Nm
must be created in that directory and it must be included into the original
.Nm
by means of the
.Fn include
function.
.Pp
.Fn include
may only be called with a relative path and with at most one directory
component.
This is by design: Kyua uses the file system structure as the layout of the
test suite definition.
Therefore, each subdirectory in a test suite must include its own
.Nm
and each
.Nm
can only descend into the
.Nm Ns s
of immediate subdirectories.
.Pp
If you need to source a
.Nm
located in disjoint parts of your file system namespace, you will have to
create a
.Sq shadow tree
using symbolic links and possibly helper
.Nm Ns s
to plug the various subdirectories together.
See the
.Sx EXAMPLES
section below for details.
.Pp
Note that each file is processed in its own Lua environment: there is no
mechanism to pass state from one file to the other.
The reason for this is that there is no such thing as a
.Dq top-level
.Nm
in a test suite: the user has to be able to run the test suite from any
directory in a given hierarchy, and this execution must not depend on files
that live in parent directories.
.Ss Top-level Kyuafile
Every system has a top directory into which test suites get installed.
The default is
.Pa /usr/tests .
Within this directory live test suites, each of which is in an independent
subdirectory.
Each subdirectory can be provided separately by independent third-party
packages.
.Pp
Kyua allows running all the installed test suites at once in order to
provide comprehensive cross-component reports.
In order to do this, there is a special file in the top directory that knows
how to inspect the subdirectories in search for other Kyuafiles and include
them.
.Pp
The
.Sx FILES
section includes more details on where this file lives.
.Ss Helper functions
The
.Sq base ,
.Sq string ,
and
.Sq table
Lua modules are fully available in the context of a
.Nm .
.Pp
The following extra functions are provided by Kyua:
.Bl -tag -width XX -offset indent
.It Ft string Fn current_kyuafile
Returns the absolute path to the current
.Nm .
.It Ft string Fn fs.basename "string path"
Returns the last component of the given path.
.It Ft string Fn fs.dirname "string path"
Returns the given path without its last component or a dot if the path has
a single component.
.It Ft bool Fn fs.exists "string path"
Checks if the given path exists.
If the path is not absolute, it is relative to the directory containing the
.Nm
in which the call to this function occurs.
.It Ft iterator Fn fs.files "string path"
Opens a directory for scanning of its entries.
The returned iterator yields an entry on each call, and the entry is simply
the filename.
If the path is not absolute, it is relative to the directory containing the
.Nm
in which the call to this function occurs.
.It Ft is_absolute Fn fs.is_absolute "string path"
Returns true if the given path is absolute; false otherwise.
.It Ft join Fn fs.join "string path" "string path"
Concatenates the two paths.
The second path cannot be absolute.
.El
.Sh FILES
.Bl -tag -width XX
.It Pa /usr/tests/Kyuafile .
Top-level
.Nm
for the current system.
.It Pa /usr/share/examples/kyua/Kyuafile.top .
Sample file to serve as a top-level
.Nm .
.El
.Sh EXAMPLES
The following
.Nm
is the simplest you can define.
It provides a test suite definition and registers a couple of different test
programs using different interfaces:
.Bd -literal -offset indent
syntax(2)
test_suite('first')
atf_test_program{name='integration_test'}
plain_test_program{name='legacy_test'}
.Ed
.Pp
The following example is a bit more elaborate.
It introduces some metadata properties to the test program definitions and
recurses into a couple of subdirectories:
.Bd -literal -offset indent
syntax(2)
test_suite('second')
plain_test_program{name='legacy_test',
allowed_architectures='amd64 i386',
required_files='/bin/ls',
timeout=30}
tap_test_program{name='privileged_test',
required_user='root'}
include('module-1/Kyuafile')
include('module-2/Kyuafile')
.Ed
.Pp
The syntax to define custom properties may be not obvious if their names
have any characters that make the property name not be a valid Lua identifier.
Dashes are just one example.
To set such properties, do something like this:
.Bd -literal -offset indent
syntax(2)
test_suite('FreeBSD')
plain_test_program{name='the_test',
['custom.FreeBSD-Bug-Id']='category/12345'}
.Ed
.Ss Connecting disjoint test suites
Now suppose you had various test suites on your file system and you would
like to connect them together so that they could be executed and treated as
a single unit.
The test suites we would like to connect live under
.Pa /usr/tests ,
.Pa /usr/local/tests
and
.Pa ~/local/tests .
.Pp
We cannot create a
.Nm
that references these because the
.Fn include
directive does not support absolute paths.
Instead, what we can do is create a shadow tree using symbolic links:
.Bd -literal -offset indent
$ mkdir ~/everything
$ ln -s /usr/tests ~/everything/system-tests
$ ln -s /usr/local/tests ~/everything/local-tests
$ ln -s ~/local/tests ~/everything/home-tests
.Ed
.Pp
And then we create an
.Pa ~/everything/Kyuafile
file to drive the execution of the integrated test suite:
.Bd -literal -offset indent
syntax(2)
test_suite('test-all-the-things')
include('system-tests/Kyuafile')
include('local-tests/Kyuafile')
include('home-tests/Kyuafile')
.Ed
.Pp
Or, simply, you could reuse the sample top-level
.Nm
to avoid having to manually craft the list of directories into which to
recurse:
.Bd -literal -offset indent
$ cp /usr/share/examples/kyua/Kyuafile.top ~/everything/Kyuafile
.Ed
.Sh SEE ALSO
.Xr kyua 1

14
usr.bin/kyua/Makefile
View File

@ -48,6 +48,20 @@ STORE= migrate_v1_v2.sql migrate_v2_v3.sql schema_v3.sql
STOREDIR= ${KYUA_STOREDIR}
.PATH: ${KYUA_SRCDIR}/store
CLEANFILES+= ${MAN}
.PATH: ${KYUA_SRCDIR}/doc
.for man in ${MAN}
${man}: ${man}.in
${SH} ${KYUA_SRCDIR}/doc/manbuild.sh \
-v "CONFDIR=${KYUA_CONFDIR}" \
-v "DOCDIR=${KYUA_DOCDIR}" \
-v "EGDIR=${KYUA_EGDIR}" \
-v "MISCDIR=${KYUA_MISCDIR}" \
-v "PACKAGE=kyua" \
-v "STOREDIR=${KYUA_STOREDIR}" \
-v "TESTSDIR=${TESTSBASE}" \
-v "VERSION=${KYUA_VERSION}" \
${.ALLSRC} ${.TARGET}
.endfor
.include <bsd.prog.mk>