$FreeBSD$
A brief how-to
--------------
Each nsswitch regression test does 2 kinds of actions:
1. It runs a series of queries and tests the correctness of results.
There are 2 basic criteria which are used for that:
- numbers must be in the correct range
- certain pointers should not be NULL
2. It makes a snapshot of the results of all queries that were made.
The idea of snapshots is to test that nsswitch-related function
calls behave equally (i.e. return same results for the same queries)
between system upgrades. When the test is executed and the snapshot is
already created, the test will compare the result of each query with
the appropriate result from the snapshot and will signal if they
differ.
In order for nsswitch tests to be as useful as possible you should use
them in the following way:
Step 1 (before upgrading your system).
Build the tests with "make" command and execute them with "prove -v"
command. If there are errors during the execution, then appropriate
nsswitch functions should be checked. Note, that errors on this state
can happen only if the particular function return incorrect data.
After the stage 1 a number of "snapshot_[test name]" files will appear
in your test folder.
Step 2 (after upgrading you system).
Rebuild the tests with "make clean; make" command and execute them
again with "prove -v" (check that "snapshot_[test name]" files
are in the same folder with tests). On this stage regression tests
will catch not only the correctness errors, but will also determine
the changes in nsswitch functions behaviour.
In case of the test failure you will get the following message:
To get more details about the error you should do the following:
Step 1. Run the test alone with debug output enabled.
Step 2. Mail the snapshot file and the debug test output to the
freebsd-current@ mailing list.
Example testing session for getpwXXX() family of functions
----------------------------------------------------------
1. make
2. prove -v ./test-getpw.t
test-getpw....1..8
ok 1 - getpwnam()
ok 2 - getpwuid()
ok 3 - getpwent()
ok 4 - getpwent() 2-pass
ok 5 - building snapshot, if needed
ok 6 - getpwnam() snapshot
ok 7 - getpwuid() snapshot
ok 8 - getpwent() snapshot
ok
All tests successful.
Files=1, Tests=8, 1 wallclock secs ( 0.00 cusr + 0.20 csys = 0.20 CPU)
3. Upgrading the system.
4. make clean; make
5. prove -v ./test-getpw.t (suppose that something has gone wrong)
test-getpw....1..8
ok 1 - getpwnam()
ok 2 - getpwuid()
ok 3 - getpwent()
ok 4 - getpwent() 2-pass
ok 5 - building snapshot, if needed
not ok 6 - getpwnam() snapshot
ok 7 - getpwuid() snapshot
not ok 8 - getpwent() snapshot
FAILED tests 6, 8
Failed 2/8 tests, 75.00% okay
Failed 1/1 test scripts, 0.00% okay. 2/8 subtests failed, 75.00% okay.
6. We see that test number 6 failed. According to get-getpw.t, this test
is executed like this:
do_test 6 'getpwnam() snapshot' '-n -s snapshot_pwd'
To determine why the test has failed, we need to run it in debug mode -
it means adding "-d" to the options list.
7. ./test-getpw -dn -s snapshot_pwd
...
testing getpwnam() with the following data:
toor:*:0:0:0::ne-again Superuser:/root::0:4831
testing correctness with the following data:
toor:*:0:0:0::Bourne-again Superuser:/root::0:4831
correct
not ok
8. Here we can see that the data from snapshot (first "toor" line) and
the data received from the getpwnam() call (second "toor" line) are
different. It is the reason why the test has failed. If you can't
(or don't want) to investigate the problem by yourself, mail
the test debug output and the snapshot file to the developers list.
Notes on using standalone nsswitch tests
----------------------------------------
All nsswitch tests have [-d] optional command line argument which enables
debug output. The debug output can be extremely helpful to determine the
cause of test failure.
In all nsswitch tests -s <file> command line argument specifies the
snapshot file. If this file doesn't exist, it would be built during
test execution. If it already exists then it will be used to check
the results of particular function calls. This argument is mostly
optional, but some tests (test-getaddr and test-getusershell) force
it to be specified.
test-gethostby and test-getaddr require the list of hostnames, that should
be queried during the test. This list must be specified via -f <file>
command line argument. Each hostname should occupy exactly one line
in the file.
Detailed tests description
--------------------------
./test-getaddr - tests the getaddrinfo() function.
Usage: test-getaddr [-d] [-46] [-s <file>] -f <file>
-d - enable debug output
-4 - force IPv4 usage
-6 - force IPv6 usage
-s - build/use specified snapshot file
-f - use specified hostnames list for testing
./test-getgr
Usage: test-getgr -nge2 [-d] [-s <file>]
-d - enable debug output
-n - test getgrnam(3)
-g - test getgrgid(3)
-e - test getgrent(3)
-2 - test getgrent(3) in 2-pass mode
-s - build/use specified snapshot file
./test-gethostby
Usage: test-gethostby -na2i [-o] [-d] [-m46] [-s <file>] -f <file>
-n - test gethostbyname2(3)
-a - test gethostbyaddr(3)
-2 - test gethostbyname2(3) results to be equal with getaddrinfo(3)
results for the similar query
-i - test gethostbyaddr(3) results to be equal with getnameinfo(3)
results for the similar query
-o - use getipnodebyname(3)/getipnodebyaddr(3) for testing instead of
gethostbyname2(3)/gethostbyaddr(3)
-d - enable debug output
-m - force IPv4-to-IPv6 mapping
-4 - force IPv4 usage
-6 - force IPv6 usage
-s - build/use specified snapshot file
-f - use specified hostnames list for testing
./test-getproto
Usage: test-getproto -nve2 [-d] [-s <file>]
-d - enable debug output
-n - test getprotobyname(3)
-v - test getprotobynumber(3)
-e - test getprotoent(3)
-2 - test getprotoent(3) in 2-pass mode
-s - build/use specified snapshot file
./test-getpw
Usage: test-getpw -nue2 [-d] [-s <file>]
-d - enable debug output
-n - test getpwnam(3)
-u - test getpwuid(3)
-e - test getpwent(3)
-2 - test getpwent(3) in 2-pass mode
-s - build/use snapshot file
./test-getrpc
Usage: test-getrpc -nve2 [-d] [-s <file>]
-d - enable debug output
-n - test getrpcbyname(3)
-v - test getrpcbynumber(3)
-e - test getrpcent(3)
-2 - test getrpcent(3) in 2-pass mode
-s - build/use specified snapshot file
./test-getserv
Usage: test-getserv -npe2 [-d] [-s <file>]
-d - enable debug output
-n - test getservbyname(3)
-p - test getservbyport(3)
-e - test getservent(3)
-2 - test getservent(3) in 2-pass mode
-s - build/use specified snapshot file
./test-getusershell
Usage: test-getusershell [-d] -s <file>
-d - enable debug output
-s - build/use specified snapshot file
