freebsd-dev/contrib/atf/doc/atf-formats.5

.\"
.\" Automated Testing Framework (atf)
.\"
.\" Copyright (c) 2007 The NetBSD Foundation, Inc.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND
.\" CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
.\" INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
.\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
.\" IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS BE LIABLE FOR ANY
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
.\" GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
.\" IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
.\" OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
.\" IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
.\"
.Dd December 20, 2011
.Dt ATF-FORMATS 5
.Os
.Sh NAME
.Nm atf-formats
.Nd machine-parseable data formats used by ATF
.Sh DESCRIPTION
This manual page describes the multiple data formats used in ATF.
These formats affect configuration files, control files and any data that
is externalized or internalized by the tools.
.Pp
Data files are always organized as follows:
.Bd -literal -offset indent
Header1: Value1            \\
    ...                    | head
HeaderN: ValueN            /
                           mandatory blank line
Free-form text contents    \\
    ...                    | body
    ...                    /
.Ed
.Pp
A file must always contain a
.Sq Content-Type
header and must always separate that header from the body with a blank
line, even if the body is empty.
.Pp
The
.Sq Content-Type
is always of the form:
.Bd -literal -offset indent
Content-Type: application/X-atf-<subtype>; version="<version>"
.Ed
.Pp
where
.Sq subtype
indicates the specific file format and
.Sq version
its format version.
This header must be the first one of the file.
.Pp
The main purpose of the
.Sq Content-Type
header, aside from determining the format used in the file, is to allow
future changes to a given format.
Whenever an incompatible change is made, the version is bumped by one.
By keeping the header in the first line, future versions may even remove
the need for such a header -- e.g. if some format was replaced by XML
files, which have their own mandatory header.
.Pp
The rest of this document details the different format types.
.Ss Format: application/X-atf-atffile, version: 1
Atffiles are logically divided into three sections:
.Bl -bullet
.It
Test programs: the list of test programs that define the test suite
described by the Atffile.
.It
Meta-data properties: these define some constant values applicable to
all the test programs defined in the file.
In some sense they define the properties that describe the test suite.
.It
Configuration variables: defaults for configuration variables that
can be overridden through configuration files or the command line.
.El
.Pp
The grammar for Atffiles is the following:
.Bd -literal -offset indent
DATA ::= ( ( CONF | PROP | TP )? COMMENT? NEWLINE )* EOF
CONF ::= 'conf:' WORD EQUAL STRING
PROP ::= 'prop:' WORD EQUAL STRING
TP ::= TPFILE | TPGLOB
TPFILE ::= 'tp: ' STRING
TPGLOB ::= 'tp-glob: ' STRING
STRING ::= WORD | '"' WORD* '"'
.Ed
.Pp
The meaning of the constructions above is:
.Bl -tag -width TPGLOBXX
.It CONF
Definition of a configuration variable.
.It PROP
Definition of a meta-data property.
.It TPFILE
Addition of a test program into the test suite.
The string is taken literally as the program's name, and this program
must exist.
.It TPGLOB
Addition of multiple test programs into the test suite.
The string is taken as a glob pattern, which may have or not have any
matches in the current directory.
.El
.Pp
An example:
.Bd -literal -offset indent
prop: test-suite = utilities

conf: unprivileged-user = nobody

tp: t_cp
tp: t_mv
tp: t_df
tp-glob: t_dir_*
.Ed
.Ss Format: application/X-atf-config, version: 1
Configuration files are very simple: they only contain a list of variable
name/variable value pairs.
Their grammar is:
.Bd -literal -offset indent
DATA ::= ( VAR? COMMENT? NEWLINE )* EOF
VAR ::= WORD EQUAL STRING
COMMENT ::= HASH WORD*
STRING ::= WORD | '"' WORD* '"'
.Ed
.Pp
An example:
.Bd -literal -offset indent
# This is the system-wide configuration file for ATF.
# The above and this line are comments placed on their own line.

var1 = this is a variable value
var2 = this is another one      # Optional comment at the end.
.Ed
.Ss Format: application/X-atf-tps, version: 3
The
.Sq application/X-atf-tps
format multiplexes the standard output, standard error and results output
streams from multiple test programs into a single data file.
This format is used by
.Xr atf-run 1
to report the execution of several test programs and is later parsed by
.Xr atf-report 1
to inform the user of this process.
It has the following grammar:
.Bd -literal -offset indent
DATA ::= INFO* TPS-COUNT TP-STANZA* INFO* EOF
INFO ::= 'info' COLON STRING COMMA STRING NEWLINE
TPS-COUNT ::= 'tps-count' COLON POSITIVE-NUMBER NEWLINE
TP-STANZA ::= TP-START TC-STANZA* TC-END
TP-START ::= 'tp-start' COLON TIMESTAMP COMMA STRING COMMA
             POSITIVE-NUMBER NEWLINE
TP-END ::= 'tc-end' COLON TIMESTAMP COMMA STRING (COMMA STRING)?
TC-STANZA ::= TC-START (TC-SO | TC-SE)* TC-END
TC-START ::= 'tc-start' COLON TIMESTAMP COMMA STRING NEWLINE
TC-SO ::= 'tc-so' COLON STRING NEWLINE
TC-SE ::= 'tc-se' COLON STRING NEWLINE
TC-END ::= 'tc-end' COLON TIMESTAMP COMMA STRING COMMA TCR NEWLINE
TCR ::= 'passed' | ('failed' | 'skipped') COMMA STRING
TIMESTAMP ::= [0-9]+.[0-9]+
.Ed
.Pp
The meaning of the constructions above is:
.Bl -tag -width TPSXCOUNTXX
.It TPS-COUNT
Indicates the number of test programs that will be executed.
There will be this exact amount of
.Sq TP-STANZA
constructions following it.
.It TP-START
Indicates the beginning of a test program.
This includes the program's name and the amount of test cases that
will follow.
.It TP-END
Indicates the completion of a test program.
This is followed by the program's name and, if the program ended
prematurely, an error message indicating the reason of its failure.
A successful execution of a test program (regardless of the status of its
test cases) must not be accompanied by any reason.
.It TC-START
Indicates the beginning of a test case.
This is accompanied by the test case's name.
.It TC-SO
Contains a text line sent to the standard output stream during the
execution of the test case.
Leading and trailing space is preserved.
.It TC-SE
Contains a text line sent to the standard error stream during the
execution of the test case.
Leading and trailing space is preserved.
.It TC-END
Indicates the completion of a test case.
This is accompanied by the test case's name, its result and the reason
associated with this result (if applicable).
.El
.Pp
An example:
.Bd -literal -offset indent
tps-count: 2
tp-start: calculator, 1324318951.838923, 2
tc-start: add, 1324318951.839101
tc-end: add, 1324318951.839500, passed
tc-start: subtract, 1324318951.840001
tc-so: 3-2 expected to return 1 but got 0
tc-end: subtract, 1324318952.000123, failed, Calculated an unexpected value
tp-end: calculator, 1324318952.002301
tp-start: files, 1, 1324318952.502348
tc-start: copy, 1324318952.508291
tc-se: could not find the cp(1) utility
tc-end: copy, 1324318953.203145, skipped
tp-end: files, 1324318953.203800
.Ed
.Sh SEE ALSO
.Xr atf 7