brooks 2af35c15f0 Import the kyua test framework.

Having kyua in the base system will simplify automated testing in CI and
eliminates bootstrapping issues on new platforms.

The build of kyua is controlled by WITH(OUT)_TESTS_SUPPORT.

Reviewed by:	emaste
Obtained from:	CheriBSD
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D24103

2020-03-23 19:01:23 +00:00

26 KiB

Raw Blame History

Major changes between releases

Changes in version 0.14

NOT RELEASED YET; STILL UNDER DEVELOPMENT.

Explicitly require C++11 language features when compiling Kyua.

Changes in version 0.13

Released on August 26th, 2016.

Fixed execution of test cases as an unprivileged user, at least under NetBSD 7.0. Kyua-level failures were probably a regression introduced in Kyua 0.12, but the underlying may have existed for much longer: test cases might have previously failed for mysterious reasons when running under an unprivileged user.
Issue #134: Fixed metadata test broken on 32-bit platforms.
Issue #139: Added per-test case start/end timestamps to all reports.
Issue #156: Fixed crashes due to the invalid handling of cleanup routine data and triggered by the reuse of PIDs in long-running Kyua instances.
Issue #159: Fixed TAP parser to ignore case while matching TODO and SKIP directives, and to also recognize Skipped.
Fixed potential crash due to a race condition in the unprogramming of timers to control test deadlines.

Changes in version 0.12

Released on November 22nd, 2015.

This is a huge release and marks a major milestone for Kyua as it finally implements a long-standing feature request: the ability to execute test cases in parallel. This is a big deal because test cases are rarely CPU-bound: running them in parallel yields much faster execution times for large test suites, allowing faster iteration of changes during development.

As an example: the FreeBSD test suite as of this date contains 3285 test cases. With sequential execution, a full test suite run takes around 12 minutes to complete, whereas on a 4-core machine with a high level of parallelism it takes a little over 1 minute.

Implementing parallel execution required rewriting most of Kyua's core and partly explains explains why there has not been a new release for over a year. The current implementation is purely subprocess-based, which works but has some limitations and has resulted in a core that is really complex and difficult to understand. Future versions will investigate the use of threads instead for a simplified programming model and additional parallelization possibilities.

Issue #2: Implemented support to execute test cases in parallel when invoking kyua test. Parallel execution is only enabled when the new parallelism configuration variable is set to a value greater than 1. The default behavior is still to run tests sequentially because some test suites contain test cases with side-effects that might fail when run in parallel. To resolve this, the new metadata property is_exclusive can be set to true on a test basis to indicate that the test must be run on its own.
Known regression: Running kyua debug on a TAP-based test program does not currently report the output in real time. The output will only be displayed once the test program completes. This is a shortcoming of the new parallel execution engine and will be resolved.
Removed the external C-based testers code in favor of the new built-in implementations. The new approach feels significantly faster than the previous one.
Fixed the handling of relative paths in the fs.* functions available in Kyuafiles. All paths are now resolved relative to the location of the caller Kyuafile. Kyuafile.top has been updated with these changes and you should update custom copies of this file with the new version.
Changed temporary directory creation to always grant search permissions on temporary directories. This is to prevent potential problems when running Kyua as root and executing test cases that require dropping privileges (as they may later be unable to use absolute paths that point inside their work directory).
The cleanup of work directories does not longer attempt to deal with mount points. If a test case mounts a file system and forgets to unmount it, the mount point will be left behind. It is now the responsibility of the test case to clean after itself. The reasons for this change are simplicity and clarity: there are many more things that a test case can do that have side-effects on the system and Kyua cannot protect against them all, so it is better to just have the test undo anything it might have done.
Improved kyua report --verbose to properly handle environment variables with continuation lines in them, and fixed the integration tests for this command to avoid false negatives.
Changed the configuration file format to accept the definition of unknown variables without declaring them local. The syntax version number remains at 2. This is to allow configuration files for newer Kyua versions to work on older Kyua versions, as there is no reason to forbid this.
Fixed stacktrace gathering with FreeBSD's ancient version of GDB. GDB 6.1.1 (circa 2004) does not have the -ex flag so we need to generate a temporary GDB script and feed it to GDB with -x instead.
Issue #136: Fixed the XML escaping in the JUnit output so that non-printable characters are properly handled when they appear in the process's stdout or stderr.
Issue #141: Improved reporting of errors triggered by sqlite3. In particular, all error messages are now tagged with their corresponding database filename and, if they are API-level errors, the name of the sqlite3 function that caused them.
Issue #144: Improved documentation on the support for custom properties in the test metadata.
Converted the INSTALL, NEWS, and README distribution documents to Markdown for better formatting online.

Changes in version 0.11

Released on October 23rd, 2014.

Added support to print the details of all test cases (metadata and their output) to report. This is via a new --verbose flag which replaces the previous --show-context.
Added support to specify the amount of physical disk space required by a test case. This is in the form of a new required_disk_space metadata property, which can also be provided by ATF test cases as require.diskspace.
Assimilated the contents of all the kyua-*-tester(1) and kyua-*-interface(7) manual pages into more relevant places. In particular, added more details on test program registration and their metadata to kyuafile(5), and added kyua-test-isolation(7) describing the isolation features of the test execution.
Assimilated the contents of all auxiliary manual pages, including kyua-build-root(7), kyua-results-files(7), kyua-test-filters(7) and kyua-test-isolation(7), into the relevant command-specific manual pages. This is for easier discoverability of relevant information when reading how specific Kyua commands work.
Issue #30: Plumbed through support to query configuration variables from ATF's test case heads. This resolves the confusing situation where test cases could only do this from their body and cleanup routines.
Issue #49: Extended report to support test case filters as command-line arguments. Combined with --verbose, this allows inspecting the details of a test case failure after execution.
Issue #55: Deprecated support for specifying test_suite overrides on a test program basis. This idiom should not be used but support for it remains in place.
Issue #72: Added caching support to the getcwd(3) test in configure so that the result can be overriden for cross-compilation purposes.
Issue #83: Changed manual page headings to include a kyua prefix in their name. This prevents some possible confusion when displaying, for example, the kyua-test manual page with a plain name of test.
Issue #84: Started passing test-suite configuration variables to plain and TAP test programs via the environment. The name of the environment variables set this way is prefixed by TEST_ENV_, so a configuration variable of the form test_suites.some_name.allow_unsafe_ops=yes in kyua.conf becomes TEST_ENV_allow_unsafe_ops=YES in the environment.
Issues #97 and #116: Fixed the build on Illumos.
Issue #102: Set TMPDIR to the test case's work directory when running the test case. If the test case happens to use the mktemp(3) family of functions (due to misunderstandings on how Kyua works or due to the reuse of legacy test code), we don't want it to easily escape the automanaged work directory.
Issue #103: Started being more liberal in the parsing of TAP test results by treating the number in ok and not ok lines as optional.
Issue #105: Started using tmpfs instead of md as a temporary file system for tests in FreeBSD so that we do not leak md(4) devices.
Issue #109: Changed the privilege dropping code to start properly dropping group privileges when unprivileged_user is set. Also fixes testers/run_test:fork_wait__unprivileged_group.
Issue #110: Changed help to display version information and clarified the purpose of the about command in its documentation.
Issue #111: Fixed crash when defining a test program in a Kyuafile that has not yet specified the test suite name.
Issue #114: Improved the kyuafile(5) manual page by clarifying the restrictions of the include() directive and by adding abundant examples.

Changes in version 0.10

Experimental version released on August 14th, 2014.

Merged kyua-cli and kyua-testers into a single kyua package.
Dropped the kyua-atf-compat package.
Issue #100: Do not try to drop privileges to unprivileged_user when we are already running as an unprivileged user. Doing so is not possible and thus causes spurious test failures when the current user is not root and the current user and unprivileged_user do not match.
Issue #79: Mention kyua.conf(5) in the See also section of kyua(1).
Issue #75: Change the rewrite__expected_signal__bad_arg test in testers/atf_result_test to use a different signal value. This is to prevent triggering a core dump that made the test fail in some platforms.

Changes in kyua-cli version 0.9

Experimental version released on August 8th, 2014.

Major changes:

The internal architecture of Kyua to record the results of test suite runs has completely changed in this release. Kyua no longer stores all the different test suite run results as different "actions" within the single store.db database. Instead, Kyua now generates a separate results file inside ~/.kyua/store/ for every test suite run.

Due to the complexity involved in the migration process and the little need for it, this is probably going to be the only release where the db-migrate command is able to convert an old store.db file to the new scheme.

Changes in more detail:

Added the report-junit command to generate JUnit XML result files. The output has been verified to work within Jenkins.
Switched to results files specific to their corresponding test suite run. The unified store.db file is now gone: kyua test creates a new results file for every invocation under ~/.kyua/store/ and the kyua report* commands are able to locate the latest file for a corresponding test suite automatically.
The db-migrate command takes an old store.db file and generates one results file for every previously-recorded action, later deleting the store.db file.
The --action flag has been removed from all commands that accepted it. This has been superseded by the tests results files.
The --store flag that many commands took has been renamed to --results-file in line with the semantical changes.
The db-exec command no longer creates an empty database when none is found. This command is now intended to run only over existing files.

Changes in kyua-testers version 0.3

Experimental version released on August 8th, 2014.

Made the testers set a "sanitized" value for the HOME environment variable where, for example, consecutive and trailing slashes have been cleared. Mac OS X has a tendency to append a trailing slash to the value of TMPDIR, which can cause third-party tests to fail if they compare ${HOME} with $(pwd).
Issues #85, #86, #90 and #92: Made the TAP parser more complete: mark test cases reported as TODO or SKIP as passed; handle skip plans; ignore lines that look like ok and not ok but aren't results; and handle test programs that report a pass but exit with a non-zero code.

Changes in kyua-cli version 0.8

Experimental version released on December 7th, 2013.

Added support for Lutok 0.4.
Issue #24: Plug the bootstrap tests back into the test suite. Fixes in kyua-testers 0.2 to isolate test cases into their own sessions should allow these to run fine.
Issue #74: Changed the kyuafile(5) parser to automatically discover existing tester interfaces. The various *_test_program() functions will now exist (or not) based on tester availability, which simplifies the addition of new testers or the selective installation of them.

Changes in kyua-testers version 0.2

Experimental version released on December 7th, 2013.

Issue #74: Added the kyua-tap-tester, a new backend to interact with test programs that comply with the Test Anything Protocol.
Issue #69: Cope with the lack of AM_PROG_AR in configure.ac, which first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04 LTS, which appears stuck in 1.11.1.
Issue #24: Improve test case isolation by confining the tests to their own session instead of just to their own process group.

Changes in kyua-cli version 0.7

Experimental version released on October 18th, 2013.

Made failures from testers more resilent. If a tester fails, the corresponding test case will be marked as broken instead of causing kyua to exit.
Added the --results-filter option to the report-html command and set its default value to skip passed results from HTML reports. This is to keep these reports more succint and to avoid generating tons of detail files that will be, in general, useless.
Switched to use Lutok 0.3 to gain compatibility with Lua 5.2.
Issue #69: Cope with the lack of AM_PROG_AR in configure.ac, which first appeared in Automake 1.11.2. Fixes a problem in Ubuntu 10.04 LTS, which appears stuck in 1.11.1.

Changes in kyua-cli version 0.6

Experimental version released on February 22nd, 2013.

Issue #36: Changed kyua help to not fail when the configuration file is bogus. Help should always work.
Issue #37: Simplified the syntax() calls in configuration and Kyuafile files to only specify the requested version instead of also the format name. The format name is implied by the file being loaded, so there is no use in the caller having to specify it. The version number of these file formats has been bumped to 2.
Issue #39: Added per-test-case metadata values to the HTML reports.
Issue #40: Rewrote the documentation as manual pages and removed the previous GNU Info document.
Issue #47: Started using the independent testers in the kyua-testers package to run the test cases. Kyua does not implement the logic to invoke test cases any more, which provides for better modularity, extensibility and robustness.
Issue #57: Added support to specify arbitrary metadata properties for test programs right from the Kyuafile. This is to make plain test programs more versatile, by allowing them to specify any of the requirements (allowed architectures, required files, etc.) supported by Kyua.
Reduced automatic screen line wrapping of messages to the help command and the output of tables by db-exec. Wrapping any other messages (specially anything going to stderr) was very annoying because it prevented natural copy/pasting of text.
Increased the granularity of the error codes returned by kyua(1) to denote different error conditions. This avoids the overload of 1 to indicate both "expected" errors from specific subcommands and unexpected errors caused by the internals of the code. The manual now correctly explain how the exit codes behave on a command basis.
Optimized the database schema to make report generation almost instantaneous.
Bumped the database schema to 2. The database now records the metadata of both test programs and test cases generically, without knowledge of their interface.
Added the db-migrate command to provide a mechanism to upgrade a database with an old schema to the current schema.
Removed the GDB build-time configuration variable. This is now part of the kyua-testers package.
Issue #31: Rewrote the Kyuafile parsing code in C++, which results in a much simpler implementation. As a side-effect, this gets rid of the external Lua files required by kyua, which in turn make the tool self-contained.
Added caching of various configure test results (particularly in those tests that need to execute a test program) so that cross-compilers can predefine the results of the tests without having to run the executables.

Changes in kyua-testers version 0.1

Experimental version released on February 19th, 2013.

This is the first public release of the kyua-testers package.

The goal of this first release is to adopt all the test case execution code of kyua-cli 0.5 and ship it as a collection of independent tester binaries. The kyua-cli package will rely on these binaries to run the tests, which provides better modularity and simplicity to the architecture of Kyua.

The code in this package is all C as opposed to the current C++ codebase of kyua-cli, which means that the overall build times of Kyua are now reduced.

Changes in kyua-cli version 0.5

Experimental version released on July 10th, 2012.

Issue #15: Added automatic stacktrace gathering of crashing test cases. This relies on GDB and is a best-effort operation.
Issue #32: Added the --build-root option to the debug, list and test commands. This allows executing test programs from a different directory than where the Kyuafile scripts live. See the Build roots section in the manual for more details.
Issue #33: Removed the kyuaify.sh script. This has been renamed to atf2kyua and moved to the kyua-atf-compat module, where it ships as a first-class utility (with a manual page and tests).
Issue #34: Changed the HTML reports to include the stdout and stderr of every test case.
Fixed the build when using a "build directory" and a clean source tree from the repository.

Changes in kyua-cli version 0.4

Experimental version released on June 6th, 2012.

Added the report-html command to generate HTML reports of the execution of any recorded action.
Changed the --output flag of the report command to only take a path to the target file, not its format. Different formats are better supported by implementing different subcommands, as the options they may receive will vary from format to format.
Added a --with-atf flag to the configure script to control whether the ATF tests get built or not. May be useful for packaging systems that do not have ATF in them yet. Disabling ATF also cuts down the build time of Kyua significantly, but with the obvious drawbacks.
Grouped kyua subcommands by topic both in the output of help and in the documentation. In general, the user needs to be aware of commands that rely on a current project and those commands that rely purely on the database to generate reports.
Made help print the descriptions of options and commands properly tabulated.
Changed most informational messages to automatically wrap on screen boundaries.
Rewrote the configuration file parsing module for extensibility. This will allow future versions of Kyua to provide additional user-facing options in the configuration file.

No syntax changes have been made, so existing configuration files (version 1) will continue to be parsed without problems. There is one little exception though: all variables under the top-level test_suites tree must be declared as strings.

Similarly, the -v and --variable flags to the command line must now carry a test_suites. prefix when referencing any variables under such tree.

Changes in kyua-cli version 0.3

Experimental version released on February 24th, 2012.

Made the test command record the results of the executed test cases into a SQLite database. As a side effect, test now supports a --store option to indicate where the database lives.
Added the report command to generate plain-text reports of the test results stored in the database. The interface of this command is certainly subject to change at this point.
Added the db-exec command to directly interact with the store database.
Issue #28: Added support for the require.memory test case property introduced in ATF 0.15.
Renamed the user-specific configuration file from ~/.kyuarc to ~/.kyua/kyua.conf for consistency with other files stored in the ~/.kyua/ subdirectory.
Switched to use Lutok instead of our own wrappers over the Lua C library. Lutok is just what used to be our own utils::lua module, but is now distributed separately.
Removed the Atffiles from the source tree. Kyua is stable enough to generate trustworthy reports, and we do not want to give the impression that atf-run / atf-report are still supported.
Enabled logging to stderr for our own test programs. This makes it slightly easier to debug problems in our own code when we get a failing test.

Changes in kyua-cli version 0.2

Experimental version released on August 24th, 2011.

The biggest change in this release is the ability for Kyua to run test programs implemented using different frameworks. What this means is that, now, a Kyua test suite can include not only ATF-based test programs, but also "legacy" (aka plain) test programs that do not use any framework. I.e. if you have tests that are simple programs that exit with 0 on success and 1 on failure, you can plug them in into a Kyua test suite.

Other than this, there have been several user-visible changes. The most important are the addition of the new config and debug subcommands to the kyua binary. The former can be used to inspect the runtime configuration of Kyua after parsing, and the latter is useful to interact with failing tests cases in order to get more data about the failure itself.

Without further ado, here comes the itemized list of changes:

Generalized the run-time engine to support executing test programs that implement different interfaces. Test programs that use the ATF libraries are just a special case of this. (Issue #18.)
Added support to the engine to run plain test programs: i.e. test programs that do not use any framework and report their pass/fail status as an exit code. This is to simplify the integration of legacy test programs into a test suite, and also to demonstrate that the run-time engine is generic enough to support different test interfaces. (Issue #18.)
Added the debug subcommand. This command allows end users to tweak the execution of a specific test case and to poke into the behavior of its execution. At the moment, all this command allows is to view the stdout and stderr of the command in real time (which the test command currently completely hides).
Added the config subcommand. This command allows the end user to inspect the current configuration variables after evaluation, without having to read through configuration files. (Issue #11.)
Removed the test_suites_var function from configuration files. This was used to set the value of test-suite-sepecific variables, but it was ugly-looking. It is now possible to use the more natural syntax test_suites.<test-suite-name>.<variable> = <value>. (Issue #11.)
Added a mechanism to disable the loading of configuration files altogether. Needed for testing purposes and for scriptability. Available by passing the --config=none flag.
Enabled detection of unused parameters and variables in the code and fixed all warnings. (Issue #23.)
Changed the behavior of "developer mode". Compiler warnings are now enabled unconditionally regardless of whether we are in developer mode or not; developer mode is now only used to perform strict warning checks and to enable assertions. Additionally, developer mode is now only automatically enabled when building from the repository, not for formal releases. (Issue #22.)
Fixed many build and portability problems to Debian sid with GCC 4.6.3 and Ubuntu 10.04.1 LTS. (Issues #20, #21, #26.)

Changes in kyua-cli version 0.1

Experimental version released on June 23rd, 2011.

This is the first public release of the kyua-cli package.

The scope of this release is to provide functional replacement for the atf-run utility included in the atf package. At this point, kyua can reliably run the NetBSD 5.99.53 test suite delivering the same results as atf-run.

The reporting facilities of this release are quite limited. There is no replacement for atf-report yet, and there is no easy way of debugging failing test programs other than running them by hand. These features will mark future milestones and therefore be part of other releases.

Be aware that this release has suffered very limited field testing. The test suite for kyua-cli is quite comprehensive, but some bugs may be left in any place.

26 KiB Raw Blame History