test: add 'match' util and use it with blobstore

Will follow this up with a doc change but want to make sure we're
all good with it first. This is meant to not only beef up
blobstore testing but provide the 'match' utilitiy for all test
cases where we are currently calling an executable and either
counting only on a return code to determine success or failure
or worse yet we're just running it and if it doesn't explode we
assume its a pass.

The 'match' util was borrowed from the PMDK folks after first
adding the "ignore" feature upstream to make it easier to use
in SPDK.  It works like this:

When the developer checks in a test they create and check in
the output of the test with two different file extensions:

.ignore: should include a string per line for output lines
that we want to totally ignore typically because they're
platform specific so the output could be different from
machine to machine.  In this case I'm ignoring all output
lines with 'DPDK' or 'EAL' or '...' in them. The first
few are obvious, the last is because the test tool will
print a varrying number of these as progress indicators.

.match: this is a copy of the output that the developer
'fixes' up by replacing platform specific output strings
with replacable tokens as described in the 'match' help.
This is where you'd want to match an entire line minus
something like a CPU count or free block count or
something. The 'ignore' feature was added simply so we
wouldn't have to edit every single line of an output
file that had DPDK or EAL in it.

Then you modify the test script to save the output and
smply run the match util providing the name of the
match file and if it fails to match the actual output
with the saved output that's been token'ized the script
will error.

The obvious advantage here is that now we can confirm all
of the output from a test executable is as we expect.

Change-Id: I701d36ee83d37b6193e16ed3171e114f814e5eb3
Signed-off-by: paul luse <paul.e.luse@intel.com>
Reviewed-on: https://review.gerrithub.io/397027
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This commit is contained in:
paul luse 2018-01-29 15:33:02 -07:00 committed by Jim Harris
parent 20cb585b0a
commit a0e22e1303
6 changed files with 478 additions and 8 deletions

View File

@ -701,7 +701,7 @@ dump_imp_open_cb(void *cb_arg, struct spdk_blob *blob, int bserrno)
} else {
cli_context->fp = fopen(cli_context->file, "r");
if (cli_context->fp == NULL) {
printf("Error in opening file\n");
printf("Error in opening file: errno %d\n", errno);
spdk_blob_close(cli_context->blob, close_cb, cli_context);
return;
}

326
test/app/match/match Executable file
View File

@ -0,0 +1,326 @@
#!/usr/bin/env perl
#
# Copyright 2014-2017, Intel Corporation
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
#
# * Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# * Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in
# the documentation and/or other materials provided with the
# distribution.
#
# * Neither the name of the copyright holder nor the names of its
# contributors may be used to endorse or promote products derived
# from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
#
# match -- compare an output file with expected results
#
# usage: match [-adoqv] [match-file]...
#
# this script compares the output from a test run, stored in a file, with
# the expected output. comparison is done line-by-line until either all
# lines compare correctly (exit code 0) or a miscompare is found (exit
# code nonzero).
#
# expected output is stored in a ".match" file, which contains a copy of
# the expected output with embedded tokens for things that should not be
# exact matches. the supported tokens are:
#
# $(N) an integer (i.e. one or more decimal digits)
# $(NC) one or more decimal digits with comma separators
# $(FP) a floating point number
# $(S) ascii string
# $(X) hex number
# $(XX) hex number prefixed with 0x
# $(W) whitespace
# $(nW) non-whitespace
# $(*) any string
# $(DD) output of a "dd" run
# $(OPT) line is optional (may be missing, matched if found)
# $(OPX) ends a contiguous list of $(OPT)...$(OPX) lines, at least
# one of which must match
#
# Additionally, if any "X.ignore" file exists, strings or phrases found per
# line in the file will be ignored if found as a substring in the
# corresponding output file (making it easy to skip entire output lines).
#
# arguments are:
#
# -a find all files of the form "X.match" in the current
# directory and match them again the corresponding file "X".
#
# -o custom output filename - only one match file can be given
#
# -d debug -- show lots of debug output
#
# -q don't print log files on mismatch
#
# -v verbose -- show every line as it is being matched
#
use strict;
use Getopt::Std;
use Encode;
use v5.16;
select STDERR;
binmode(STDOUT, ":utf8");
binmode(STDERR, ":utf8");
my $Me = $0;
$Me =~ s,.*/,,;
our ($opt_a, $opt_d, $opt_q, $opt_v, $opt_o);
$SIG{HUP} = $SIG{INT} = $SIG{TERM} = $SIG{__DIE__} = sub {
die @_ if $^S;
my $errstr = shift;
die "FAIL: $Me: $errstr";
};
sub usage {
my $msg = shift;
warn "$Me: $msg\n" if $msg;
warn "Usage: $Me [-adqv] [match-file]...\n";
warn " or: $Me [-dqv] -o output-file match-file...\n";
exit 1;
}
getopts('adoqv') or usage;
my %match2file;
if ($opt_a) {
usage("-a and filename arguments are mutually exclusive")
if $#ARGV != -1;
opendir(DIR, '.') or die "opendir: .: $!\n";
my @matchfiles = grep { /(.*)\.match$/ && -f $1 } readdir(DIR);
closedir(DIR);
die "no files found to process\n" unless @matchfiles;
foreach my $mfile (@matchfiles) {
die "$mfile: $!\n" unless open(F, $mfile);
close(F);
my $ofile = $mfile;
$ofile =~ s/\.match$//;
die "$mfile found but cannot open $ofile: $!\n"
unless open(F, $ofile);
close(F);
$match2file{$mfile} = $ofile;
}
} elsif ($opt_o) {
usage("-o argument requires two paths") if $#ARGV != 1;
$match2file{$ARGV[1]} = $ARGV[0];
} else {
usage("no match-file arguments found") if $#ARGV == -1;
# to improve the failure case, check all filename args exist and
# are provided in pairs now, before going through and processing them
foreach my $mfile (@ARGV) {
my $ofile = $mfile;
usage("$mfile: not a .match file") unless
$ofile =~ s/\.match$//;
usage("$mfile: $!") unless open(F, $mfile);
close(F);
usage("$ofile: $!") unless open(F, $ofile);
close(F);
$match2file{$mfile} = $ofile;
}
}
my $mfile;
my $ofile;
my $ifile;
print "Files to be processed:\n" if $opt_v;
foreach $mfile (sort keys %match2file) {
$ofile = $match2file{$mfile};
$ifile = $ofile . ".ignore";
$ifile = undef unless (-f $ifile);
if ($opt_v) {
print " match-file \"$mfile\" output-file \"$ofile\"";
if ($ifile) {
print " ignore-file $ifile\n";
} else {
print "\n";
}
}
match($mfile, $ofile, $ifile);
}
exit 0;
#
# strip_it - user can optionally ignore lines from files that contain
# any number of substrings listed in a file called "X.ignore" where X
# is the name of the output file.
#
sub strip_it {
my ($ifile, $file, $input) = @_;
# if there is no ignore file just return unaltered input
return $input unless $ifile;
my @lines_in = split /^/, $input;
my $output;
my $line_in;
my @i_file = split /^/, snarf($ifile);
my $i_line;
my $ignore_it = 0;
foreach $line_in (@lines_in) {
my @i_lines = @i_file;
foreach $i_line (@i_lines) {
chop($i_line);
if (index($line_in, $i_line) != -1) {
$ignore_it = 1;
if ($opt_v) {
print "Ignoring (from $file): $line_in";
}
}
}
if ($ignore_it == 0) {
$output .= $line_in;
}
$ignore_it = 0;
}
return $output;
}
#
# match -- process a match-file, output-file pair
#
sub match {
my ($mfile, $ofile, $ifile) = @_;
my $pat;
my $output = snarf($ofile);
$output = strip_it($ifile, $ofile, $output);
my $all_lines = $output;
my $line_pat = 0;
my $line_out = 0;
my $opt = 0;
my $opx = 0;
my $opt_found = 0;
my $fstr = snarf($mfile);
$fstr = strip_it($ifile, $mfile, $fstr);
for (split /^/, $fstr) {
$pat = $_;
$line_pat++;
$line_out++;
s/([*+?|{}.\\^\$\[()])/\\$1/g;
s/\\\$\\\(FP\\\)/[-+]?\\d*\\.?\\d+([eE][-+]?\\d+)?/g;
s/\\\$\\\(N\\\)/[-+]?\\d+/g;
s/\\\$\\\(NC\\\)/[-+]?\\d+(,[0-9]+)*/g;
s/\\\$\\\(\\\*\\\)/\\p{Print}*/g;
s/\\\$\\\(S\\\)/\\P{IsC}+/g;
s/\\\$\\\(X\\\)/\\p{XPosixXDigit}+/g;
s/\\\$\\\(XX\\\)/0x\\p{XPosixXDigit}+/g;
s/\\\$\\\(W\\\)/\\p{Blank}*/g;
s/\\\$\\\(nW\\\)/\\p{Graph}*/g;
s/\\\$\\\(DD\\\)/\\d+\\+\\d+ records in\n\\d+\\+\\d+ records out\n\\d+ bytes \\\(\\d+ .B\\\) copied, [.0-9e-]+[^,]*, [.0-9]+ .B.s/g;
if (s/\\\$\\\(OPT\\\)//) {
$opt = 1;
} elsif (s/\\\$\\\(OPX\\\)//) {
$opx = 1;
} else {
$opt_found = 0;
}
if ($opt_v) {
my @lines = split /\n/, $output;
my $line;
if (@lines) {
$line = $lines[0];
} else {
$line = "[EOF]";
}
printf("%s:%-3d %s%s:%-3d %s\n", $mfile, $line_pat, $pat, $ofile, $line_out, $line);
}
print " => /$_/\n" if $opt_d;
print " [$output]\n" if $opt_d;
unless ($output =~ s/^$_//) {
if ($opt || ($opx && $opt_found)) {
printf("%s:%-3d [skipping optional line]\n", $ofile, $line_out) if $opt_v;
$line_out--;
$opt = 0;
} else {
if (!$opt_v) {
if ($opt_q) {
print "[MATCHING FAILED]\n";
} else {
print "[MATCHING FAILED, COMPLETE FILE ($ofile) BELOW]\n$all_lines\n[EOF]\n";
}
$opt_v = 1;
match($mfile, $ofile);
}
die "$mfile:$line_pat did not match pattern\n";
}
} elsif ($opt) {
$opt_found = 1;
}
$opx = 0;
}
if ($output ne '') {
if (!$opt_v) {
if ($opt_q) {
print "[MATCHING FAILED]\n";
} else {
print "[MATCHING FAILED, COMPLETE FILE ($ofile) BELOW]\n$all_lines\n[EOF]\n";
}
}
# make it a little more print-friendly...
$output =~ s/\n/\\n/g;
die "line $line_pat: unexpected output: \"$output\"\n";
}
}
#
# snarf -- slurp an entire file into memory
#
sub snarf {
my ($file) = @_;
my $fh;
open($fh, '<', $file) or die "$file $!\n";
local $/;
$_ = <$fh>;
close $fh;
# check known encodings or die
my $decoded;
my @encodings = ("UTF-8", "UTF-16", "UTF-16LE", "UTF-16BE");
foreach my $enc (@encodings) {
eval { $decoded = decode( $enc, $_, Encode::FB_CROAK ) };
if (!$@) {
$decoded =~ s/\R/\n/g;
return $decoded;
}
}
die "$Me: ERROR: Unknown file encoding";
}

View File

@ -11,7 +11,6 @@ set -xe
testdir=$(readlink -f $(dirname $0))
rootdir=$(readlink -f $testdir/../..)
source $rootdir/scripts/autotest_common.sh
timing_enter blobstore
set -e
@ -19,9 +18,24 @@ set -e
# Nvme0 target configuration
$rootdir/scripts/gen_nvme.sh > $testdir/blobcli.conf
$rootdir/examples/blob/cli/blobcli -c $testdir/blobcli.conf -b Nvme0n1 -T $testdir/test.bs
# generate random data file for import/export diff
dd if=/dev/urandom of=$testdir/test.pattern bs=1M count=1
cd $testdir
$rootdir/examples/blob/cli/blobcli -c $testdir/blobcli.conf -b Nvme0n1 -T $testdir/test.bs > $testdir/btest.out
cd -
# the tool leaves some trailing whitespaces that we need to strip out
sed -i 's/[[:space:]]*$//' $testdir/btest.out
# the test script will import the test pattern generated by dd and then export
# it to a file so we can compare and confirm basic read and write
$rootdir/test/app/match/match -v $testdir/btest.out.match
diff $testdir/test.pattern $testdir/test.pattern.blob
rm -rf $testdir/btest.out
rm -rf $testdir/blobcli.conf
rm -rf ./*.blob
rm -rf $testdir/*.blob
rm -rf $testdir/test.pattern
timing_exit blobstore

View File

@ -0,0 +1,3 @@
DPDK
EAL
....

View File

@ -0,0 +1,123 @@
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: blobcli -c 0x1 --file-prefix=spdk_pid5072 ]
EAL: Probing VFIO support...
EAL: PCI device 0000:00:04.0 on NUMA socket 0
EAL: probe driver: 8086:2f20 spdk_ioat
EAL: PCI device 0000:00:04.1 on NUMA socket 0
EAL: probe driver: 8086:2f21 spdk_ioat
EAL: PCI device 0000:00:04.2 on NUMA socket 0
EAL: probe driver: 8086:2f22 spdk_ioat
EAL: PCI device 0000:00:04.3 on NUMA socket 0
EAL: probe driver: 8086:2f23 spdk_ioat
EAL: PCI device 0000:00:04.4 on NUMA socket 0
EAL: probe driver: 8086:2f24 spdk_ioat
EAL: PCI device 0000:00:04.5 on NUMA socket 0
EAL: probe driver: 8086:2f25 spdk_ioat
EAL: PCI device 0000:00:04.6 on NUMA socket 0
EAL: probe driver: 8086:2f26 spdk_ioat
EAL: PCI device 0000:00:04.7 on NUMA socket 0
EAL: probe driver: 8086:2f27 spdk_ioat
EAL: PCI device 0000:80:04.0 on NUMA socket 1
EAL: probe driver: 8086:2f20 spdk_ioat
EAL: PCI device 0000:80:04.1 on NUMA socket 1
EAL: probe driver: 8086:2f21 spdk_ioat
EAL: PCI device 0000:80:04.2 on NUMA socket 1
EAL: probe driver: 8086:2f22 spdk_ioat
EAL: PCI device 0000:80:04.3 on NUMA socket 1
EAL: probe driver: 8086:2f23 spdk_ioat
EAL: PCI device 0000:80:04.4 on NUMA socket 1
EAL: probe driver: 8086:2f24 spdk_ioat
EAL: PCI device 0000:80:04.5 on NUMA socket 1
EAL: probe driver: 8086:2f25 spdk_ioat
EAL: PCI device 0000:80:04.6 on NUMA socket 1
EAL: probe driver: 8086:2f26 spdk_ioat
EAL: PCI device 0000:80:04.7 on NUMA socket 1
EAL: probe driver: 8086:2f27 spdk_ioat
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL: probe driver: 8086:953 spdk_nvme
SCRIPT NOW PROCESSING: -i
Init blobstore using bdev Product Name: NVMe disk
blobstore init'd: ($(XX))
SCRIPT NOW PROCESSING: -l bdevs
List bdevs:
bdev Name: Nvme0n1
bdev Product Name: NVMe disk
SCRIPT NOW PROCESSING: -n 1
New blob id 4294967296
blob now has USED clusters of 1
SCRIPT NOW PROCESSING: -p $B0
Super Blob ID has been set.
SCRIPT NOW PROCESSING: -n 1
New blob id 4294967297
blob now has USED clusters of 1
SCRIPT NOW PROCESSING: -m $B1 test.pattern
Working...............................................................................................................................................................................................................................................................
Blob import complete (from test.pattern).
SCRIPT NOW PROCESSING: -d $B1 test.pattern.blob
Working................................................................................................................................................................................................................................................................
File write complete (to test.pattern.blob).
SCRIPT NOW PROCESSING: -x $B1 key val
Xattr has been set.
SCRIPT NOW PROCESSING: -s bs
Blobstore Public Info:
Using bdev Product Name: NVMe disk
API Version: $(N)
super blob ID: 4294967296
page size: 4096
cluster size: 1048576
# free clusters: $(N)
blobstore type:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Blobstore Private Info:
Metadata start (pages): $(N)
Metadata length (pages): $(N)
SCRIPT NOW PROCESSING: -s $B1
Blob Public Info:
blob ID: 4294967297
# of clusters: 1
# of bytes: 1048576
# of pages: 256
# of xattrs: 1
xattrs:
(0) Name:key
(0) Value:
00000000 76 61 6c 00 val.
Blob Private Info:
state: CLEAN
open ref count: 1
SCRIPT NOW PROCESSING: -r $B1 key
Xattr has been removed.
SCRIPT NOW PROCESSING: -s bs
Blobstore Public Info:
Using bdev Product Name: NVMe disk
API Version: 3
super blob ID: 4294967296
page size: 4096
cluster size: 1048576
# free clusters: $(N)
blobstore type:
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Blobstore Private Info:
Metadata start (pages): $(N)
Metadata length (pages): $(N)
SCRIPT NOW PROCESSING: -X

View File

@ -1,8 +1,12 @@
-i
-l bdevs
-n 1
-s bs
-p $B0
-d $B0 M.blob
-x $B0 b0key boval
-r $B0 b0key
-n 1
-m $B1 test.pattern
-d $B1 test.pattern.blob
-x $B1 key val
-s bs
-s $B1
-r $B1 key
-s bs