Commit Graph

38 Commits

Author SHA1 Message Date
Enji Cooper
430f7286a5 Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed
after r298107

Summary of changes:

- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
  namespacing is kept with FILES appropriately, and that this shouldn't need
  to be repeated if the namespace changes -- only the definition of PACKAGE
  needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
  `tests`. In the event we get to the point where things can be split up
  enough in the base system, it would make more sense to group the tests
  with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
  previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
  bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
  ${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
  and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)

Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.

MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division
2016-05-04 23:20:53 +00:00
Glen Barber
7d536dc855 MFH
Sponsored by:	The FreeBSD Foundation
2016-03-10 21:16:01 +00:00
Bryan Drewery
f68f6b3dd6 Remove things set already by bsd.progs.mk.
MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2016-03-09 03:22:17 +00:00
Glen Barber
52259a98ad MFH
Sponsored by:	The FreeBSD Foundation
2016-03-02 16:14:46 +00:00
Bryan Drewery
342863e712 Move PROGS logic to proper place and remove redundant and unneeded logic.
- bsd.progs.mk is safe to include regardless of PROGS/PROGS_CXX/SCRIPTS
  being set.
- bsd.progs.mk includes bsd.prog.mk always and will bring in
  bsd.files.mk and bsd.obj.mk.
- DIRDEPS_BUILD: There's no need for _SKIP_BUILD or _SKIP_STAGING as the
  PROGS were implicitly being built by the staging dependency anyway.  This
  was also preventing other objects from building if they were not part of
  the staging sets.
- DIRDEPS_BUILD: This fixes PROGS without bsd.test.mk not staging.

MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2016-02-26 22:13:52 +00:00
Glen Barber
72c3aa02dc MFH
Sponsored by:	The FreeBSD Foundation
2016-02-18 00:37:58 +00:00
Bryan Drewery
d3157f0915 Test directories can build in parallel fine.
Sponsored by:	EMC / Isilon Storage Division
2016-02-16 02:13:59 +00:00
Glen Barber
bbb51924bb MFH
Sponsored by:	The FreeBSD Foundation
2016-02-08 12:16:01 +00:00
Enji Cooper
71b7fa1252 Simplify running the FreeBSD test suite
Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.

`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.

Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.

Document `make check` and `make checkworld` in build(7).

Other minor changes:

- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
  `make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
  not found

The MFC is [partly] contingent on other build related changes being MFCed.

Differential Revision: https://reviews.freebsd.org/D4406
MFC after: 2 months
X-MFC to: stable/10
Relnotes: yes
Reviewed by: bdrewery, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division
2016-02-07 18:40:04 +00:00
Glen Barber
b626f5a73a MFH r289384-r293170
Sponsored by:	The FreeBSD Foundation
2016-01-04 19:19:48 +00:00
Enji Cooper
26dfa13578 - Use LOCALBASE instead of KYUA_PREFIX for the --prefix to kyua(1)
- Use LOCALBASE instead of hardcoding /usr/local for perl

MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff)
Reviewed by: emaste, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division
2015-12-20 06:26:15 +00:00
Glen Barber
324fd1ce05 MFH to r289370
Sponsored by:	The FreeBSD Foundation
2015-10-15 17:36:56 +00:00
Enji Cooper
da3110ed56 Default TESTSDIR to /usr/tests/${RELDIR:H}
When run from bin/ls/tests, for example, the value of TESTSDIR would be
${TESTSBASE}/${RELDIR:H} -> /usr/tests/bin/ls/tests/.. ->
/usr/tests/bin/ls

Document the new behavior in bsd.README.

While here, also document TESTSBASE

Relnotes: yes
Differential Revision: D1022
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division
2015-10-11 21:29:24 +00:00
Baptiste Daroussin
5a2b666ce5 Merge from head 2015-10-01 09:36:43 +00:00
Bryan Drewery
abb02fa2f9 Fix most cases of bsd.progs.mk running duplicate or missing commands.
This mostly fixes an interaction with bsd.test.mk with PROGS and SCRIPTS.
This was most notable with 'make clean' and 'make install', which r281055
and r272055 attempted to address but were inadequate.

It also addresses similar issues in bsd.progs.mk when not using bsd.test.mk.

This also fixes cases of NOT running commands in the parent when using
bsd.progs.mk:
  - 'make clean' was not run for the main process for Makefiles which had both
    FILES and SUBDIR but no PROGS or SCRIPTS.  This usually was just a
    leftover Kyuafile.auto.  One such example is usr.bin/bmake/tests/sysmk/t1/2.
  - 'make obj' was not running in the current directory with bsd.test.mk due
    to early inclusion of bsd.subdir.mk.  This was not really a problem due to
    the SUBDIRS using 'mkdir -p' for their objdirs.

There were subtle bugs causing this wrong behavior:
  1. bsd.progs.mk needs to set SCRIPTS to empty when recursing to avoid
     the sub-makes from installing, cleaning or building the SCRIPTS;
     only the parent make should be doing this.  r281055 effectively did
     the same but wasn't enough.
  2. CLEANFILES may contain (especially from *.test.mk) files which only
     the parent should clean, such as from FILES and SCRIPTS.  To resolve
     sub-makes also cleaning these, reset CLEANFILES and CLEANDIRS in the
     children before including bsd.prog.mk.  A tempting alternative would be
     to only handle CLEANFILES in the parent but then the child bsd.prog.mk
     CLEANFILES of per-PROGS wouldn't be setup.
  3. bsd.subdir.mk was included too soon in bsd.test.mk.  It needs to be
     included after bsd.prog.mk as the SCRIPTS logic is short-circuitted if
     'install:' is already defined (which bsd.subdir.mk does).  There is
     actually no need to include bsd.subdir.mk from bsd.test.mk as bsd.prog.mk
     and bsd.obj.mk will do so in the proper order.  The description in r257095
     covers this for FILES and was fixed differently, though changing the
     handling of target(install) in bsd.prog.mk may make sense after more
     research.
  4. bsd.progs.mk had extra logic to handle recursing SCRIPTS if PROGS was
     empty, which isn't its business to be doing.  SCRIPTS is handled fine
     by bsd.prog.mk.  This mostly reverts and reworks the fix in r259209 and
     partially reverts r272055.
  5. bsd.progs.mk has no need to depend 'all:' on SCRIPTS and FILES.  These
     are handled by bsd.prog.mk/bsd.files.mk fine.  This also partially reverts
     r272055.
  6. bsd.progs.mk was not drop-in safe for bsd.prog.mk.  Move the PROGS
     check from r273186 to allow it to be used safely.

Specific tested cases:
  SCRIPTS:no PROGS:no FILES:yes SUBDIR:yes
    usr.bin/bmake/tests/sysmk/t1/2

  SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no
    usr.bin/bmake/tests/sysmk/t1/2/1

  SCRIPTS:yes PROGS:yes FILES:yes SUBDIR:yes
    lib/libthr/tests

  SCRIPTS:yes PROGS:no FILES:yes SUBDIR:no
    usr.bin/yacc/tests
    libexec/atf/atf-sh/tests

A full buildworld/installworld/clean comparison with mtree was also done.
The only relevant difference was the new fixed behavior of removing
Kyuafile.auto from the objdir in 'clean'.

Converting SCRIPTS to be a special case FILES group will make this less
fragile and is being explored.

One known remaining issue is 'cleandepend' removing the tags files for
every recursive call.

Note that the 'make clean' command runs for the CURDIR last, which can make
it appear to run multiple times when cleaning in tests/, but each command is
for a SUBDIR returning up the chain.  This is purely bsd.subdir.mk behavior.

PR:		191055
PR:		191955
MFC after:	2 weeks
Sponsored by:	EMC / Isilon Storage Division
2015-09-23 23:20:49 +00:00
Baptiste Daroussin
08796a78ee Move tests into a single FreeBSD-tests package 2015-06-16 21:07:07 +00:00
Enji Cooper
c4af37bb9a Ensure TESTSDIR is defined before bsd.test.mk is .include'd
MFC after: 1 week
2015-06-15 07:13:23 +00:00
Simon J. Gerraty
9268022b74 Merge from head@274682 2014-11-19 01:07:58 +00:00
Enji Cooper
05f0ff5a30 Fix the logic inversion in the previous commit by ensuring that the matched
expression (:M) is empty, not the not matched (:N) is empty. The former case
means we have not found the TEST_SUBDIR value in SUBDIR

Reported by: rodrigc
X-MFC with: r273803
Pointyhat to: me (did not use a clean install root)
Sponsored by: EMC / Isilon Storage Division
2014-10-29 04:32:46 +00:00
Enji Cooper
9aa9738938 Filter out TESTS_SUBDIRS already added to SUBDIR instead of blindly
appending the TESTS_SUBDIRS variable to SUBDIR

Duplicate directory entries can cause unexpected side effects, like
installing the same files multiple times. This can be easily
reproduced via the following testcase prior to this commit:

  SUBDIR= dir
  TESTS_SUBDIRS+= dir

  .include <bsd.test.mk>

Sponsored by: EMC / Isilon Storage Division
2014-10-28 23:01:09 +00:00
Simon J. Gerraty
ee7b0571c2 Merge head from 7/28 2014-08-19 06:50:54 +00:00
Julio Merino
e8a34402f2 Put the test suite in its own tests.txz distribution file.
Force all the contents of /usr/tests to go into a separate distribution
file so that users of binary releases can easily choose to not install it.

To make this possible, we need two fixes:
- bsd.subdir.mk needs to properly honor NO_SUBDIR in all cases so that we
  do not recurse into 'tests' subdirectories when we needn't.  Otherwise,
  we end up with some Kyuafiles in base.txz.
- etc/Makefile needs to skip installing tests in its 'distribute' target
  so that a Kyuafile doesn't leak into base.txz.

Approved by:	gjb
2014-06-10 17:04:30 +00:00
Julio Merino
ada17d7bde Ensure files are created during the build when using bsd.subdir.mk.
When FILES is defined in a Makefile that _also_ includes bsd.subdir.mk, the
build of the files (if any) was not properly triggered during the build
stage.  This was because bsd.files.mk did not define the buildfiles target
if it was already defined... and bsd.subdir.mk defined this target on its
own, thus causing a conflict.

Fix this by unconditionally defining buildfiles from bsd.files.mk; this is
safe because nothing else in the tree needs to redefine this and because the
target itself contains no commands: all it does is define dependencies.
Also ensure that bsd.files.mk is always pulled in by bsd.test.mk regardless
of what bsd.prog.mk does.

These fixes allow "make installworld" to run cleanly on a system with
read-only src and obj trees.

This is "make tinderbox" clean.

Reviewed by:	imp
Obtained from:	jilles
2014-06-09 14:36:49 +00:00
Warner Losh
c115b8184e Remove last two NO_MAN= in the tree. In both of these cases, MAN= is
what is needed.
2014-05-10 16:38:37 +00:00
Simon J. Gerraty
3b8f084595 Merge head 2014-04-28 07:50:45 +00:00
Warner Losh
40b86d77c7 The proper way to request no man pages currently is NO_MAN=xxx. Use it
in preference to the user WITHOUT_MAN knob, which should never be set
in normal src Makefiles.
2014-04-05 17:54:36 +00:00
Julio Merino
a784098c10 Move FreeBSD Test Suite-specific code to a suite.test.mk file.
The new suite.test.mk file contains all the logic needed to install test
programs under /usr/tests/ and to support Kyua as the run-time engine.
This file is included by default by bsd.test.mk so Makefiles do not need
to care about its existence.

Specific Makefiles can define NOT_FOR_TEST_SUITE to indicate that whatever
test programs they are building are not supposed to be installed under
/usr/tests/ nor run by Kyua.  (The effect of passing this setting is that
suite.test.mk is simply not included.)

NOT_FOR_TEST_SUITE should never be used by Makefiles in the base system.
This functionality is provided so that third-parties can hook in their
own test code, with different semantics, if they wish.  This was asked
for by sjg@.
2014-03-14 12:52:55 +00:00
Julio Merino
f5fd950e35 Make bsd.test.mk the only public mk fragment for the building of tests.
Change {atf,plain,tap}.test.mk to be internal implementation details of
bsd.test.mk.  Makefiles that build tests should now only include bsd.test.mk
and declaratively specify what they want to build, without worrying about
the internal implementation of the mk files.

The reason for this change is to permit building test programs of different
interfaces from a single directory, which is something I had a need for
while porting tests over from src/tools/regression/.

Additionally, this change makes it possible to perform some other requested
changes to bsd.test.mk in an easier manner.  Coming soon.
2014-03-14 08:56:19 +00:00
Julio Merino
8c465f5880 Support defining test program metadata from the Makefiles.
Introduce a new, per-test-program TEST_METADATA.<program> variable that
contains a list of key/value paris describing metadata properties for
that test program.  These properties are later written into the
auto-generated Kyuafile when using the KYUAFILE=auto functionality.

This is to avoid having to supply hand-crafted Kyuafiles when the needs
for metadata overrides are trivial.

While doing this, and because I am documenting TEST_METADATA, take the
chance to document the TEST_INTERFACE setting as well.

MFC after:	5 days
2014-01-14 18:32:47 +00:00
Julio Merino
d7efee230d Allow tests to provide a Kyuafile when they relied on auto-generation.
When generating a Kyuafile in the KYUAFILE=auto case, use a filename
that is unlikely to clash with the filename used by explicitly-provided
Kyuafiles.

This allows a Makefile to set KYUAFILE=yes and provide a Kyuafile in
the same directory when such Makefile was previously relying on
KYUAFILE=auto.

Fixes issues with new Kyuafiles not being picked up in NO_CLEAN builds
(although manual intervention is required once, unfortunately, as
described in UPDATING).

Reviewed by:	sjg
MFC after:	1 week
2014-01-10 10:36:14 +00:00
Rui Paulo
5e6f3a8ed7 Move the TESTSBASE definition to bsd.own.mk.
We need to be able to reference the value of TESTSBASE without requiring
the inclusion of bsd.test.mk (e.g. in etc/Makefile), so move its definition
to the more generic bsd.own.mk.

Submitted by:	Julio Merino jmmv google.com
Reviewed by:	sjg
MFC after:	2 weeks
2013-10-25 05:12:31 +00:00
Rui Paulo
7bb2d4b324 Plug kyua into the 'test' target.
If kyua exists in KYUA_PREFIX, the test target is automatically
defined to use it for the execution of test programs.

Submitted by:	Julio Merino jmmv google.com
MFC after:	2 weeks
2013-10-19 06:52:06 +00:00
Rui Paulo
30cc088dc0 Plug atf-run into the 'test' target.
If atf-run exists in ATF_PREFIX and if ALLOW_DEPRECATED_ATF_TOOLS has
been set to yes, the test target is automatically defined to use it
for the execution of test programs.

Submitted by:	Julio Merino jmmv google.com
MFC after:	2 weeks
2013-10-19 06:51:34 +00:00
Rui Paulo
25b6a535fa Add the automatic generation of Kyuafile files.
These files are generated from bsd.test.mk because kyua is able to run
test programs implemented using different libraries/frameworks.  In
order to make this possible, this change also extends the various
*.test.mk file to explicitly indicate the interface of every test
program.

Submitted by:	Julio Merino jmmv google.com
MFC after:	2 weeks
2013-10-19 06:50:56 +00:00
Rui Paulo
26ea29af54 Add the automatic generation of Atffile files.
These are only used by the deprecated atf-run and atf-report tools.
Generating them is easy and provides a mechanism for people to
experiment with these tools if they wish.

But, because these tools and files are deprecated, doing this only
happens if the user has explicitly set ALLOW_DEPRECATED_ATF_TOOLS
to yes.

Submitted by:	Julio Merino jmmv google.com
MFC after:	2 weeks
2013-10-19 06:50:17 +00:00
Rui Paulo
d4a14c8563 Clearly split the logic to build ATF and plain tests apart.
This change introduces a new plain.test.mk file that provides the build
infrastructure to build test programs that don't use any framework.
Most of the code previously in bsd.test.mk moves to plain.test.mk and
atf.test.mk is extended with the missing pieces.

In doing so, this change pushes all test program building logic to the
various *.test.mk files instead of trying to reuse some tiny bits.
In fact, this attempt to reuse some definitions makes the code harder
to read and harder to extend.

The clear benefit of this is that the interface of bsd.test.mk is now
clearly delimited.

Submitted by:	Julio Merino jmmv google.com
MFC after:	2 weeks
2013-10-19 06:48:49 +00:00
Simon J. Gerraty
6f5639e3f1 Sync-up with what's being use in Juniper for building ATF in meta mode. 2013-10-12 23:44:50 +00:00
Simon J. Gerraty
50c5d27b26 Enable ATF testing.
Submitted by:	Garrett Cooper
Approved by:	marcel (mentor)
2012-11-07 22:02:02 +00:00