Commit Graph

575 Commits

Author SHA1 Message Date
Mariusz Zaborski
fd631bcd95 procdesc: fix reparenting when the debugger is attached
The process is reparented to the debugger while it is attached.
  B          B
 /   ---->   |
A          A D

Every time when the process is reparented, it is added to the orphan list
of the previous parent:

A->orphan = B
D->orphan = NULL

When the A process will close the process descriptor to the B process,
the B process will be reparented to the init process.
  B            B - init
  |   ---->
A D          A   D

A->orphan = B
D->orphan = B

In this scenario, the B process is in the orphan list of A and D.

When the last process descriptor is closed instead of reparenting
it to the reaper let it stay with the debugger process and set
our previews parent to the reaper.

Add test case for this situation.
Notice that without this patch the kernel will crash with this test case:
panic: orphan 0xfffff8000e990530 of 0xfffff8000e990000 has unexpected oppid 1

Reviewed by:	markj, kib
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D20361
2019-08-05 20:15:46 +00:00
Tom Jones
f97a8a3615 Add common firewall test suite
Add a common test suite for the firewalls included in the base system. The test
suite allows common test infrastructure to test pf, ipfw and ipf firewalls from
test files containing the setup for all three firewalls.

Add the pass block test for pf, ipfw and ipf. The pass block test checks the
allow/deny functionality of the firewalls tested.

Submitted by:   Ahsan Barkati
Sponsored by:   Google, Inc. (GSoC 2019)
Reviewed by:    kp
Approved by:    bz (co-mentor)
MFC after:      2 weeks
Differential Revision: https://reviews.freebsd.org/D21065
2019-08-05 11:47:34 +00:00
Li-Wen Hsu
9777e3544a Only skip test cases sometimes failing in CI when they are running in CI
Suggested by:	jhb
Sponsored by:	The FreeBSD Foundation
2019-08-01 18:19:16 +00:00
Li-Wen Hsu
1ab93d1f23 Temporarily skip flakey test case
sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger

PR:		239425
Sponsored by:	The FreeBSD Foundation
2019-07-24 17:41:40 +00:00
Li-Wen Hsu
c2dc497a38 Temporarily skip flakey test case
sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger

PR:		239399
Sponsored by:	The FreeBSD Foundation
2019-07-23 09:39:27 +00:00
Li-Wen Hsu
ea24861d5e Temporarily skip flakey test case
sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger

PR:		239397
Sponsored by:	The FreeBSD Foundation
2019-07-23 09:19:58 +00:00
Li-Wen Hsu
7d1f74716c Temporarily skip flakey test case
sys.kern.ptrace_test.ptrace__PT_KILL_competing_stop

PR:		220841
Sponsored by:	The FreeBSD Foundation
2019-07-23 07:56:42 +00:00
Li-Wen Hsu
4f6d74c9c4 Temporarily skip sys.netpfil.pf.forward.{v4,v6} and sys.netpfil.pf.set_tos.v4
on i386 as they are flakey on it

PR:		239380
Sponsored by:	The FreeBSD Foundation
2019-07-22 18:54:26 +00:00
Li-Wen Hsu
63b0609c12 Fix URL.
Sponsored by:	The FreeBSD Foundation
2019-07-22 18:43:46 +00:00
Li-Wen Hsu
37ba9b348b Temporarily skip flakey test case
sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger

PR:		239292
Sponsored by:	The FreeBSD Foundation
2019-07-22 10:37:56 +00:00
John Baldwin
32451fb9fc Add ptrace op PT_GET_SC_RET.
This ptrace operation returns a structure containing the error and
return values from the current system call.  It is only valid when a
thread is stopped during a system call exit (PL_FLAG_SCX is set).

The sr_error member holds the error value from the system call.  Note
that this error value is the native FreeBSD error value that has _not_
been translated to an ABI-specific error value similar to the values
logged to ktrace.

If sr_error is zero, then the return values of the system call will be
set in sr_retval[0] and sr_retval[1].

Reviewed by:	kib
MFC after:	1 month
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D20901
2019-07-15 21:48:02 +00:00
John Baldwin
c8ea87310c Add a test for PT_GET_SC_ARGS.
Reviewed by:	kib
MFC after:	1 month
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D20899
2019-07-15 21:26:55 +00:00
Li-Wen Hsu
1db8307b66 Correct definitions in sys.opencrypto.runtests.main for 32bit platform
Reviewed by:	cem, jhb
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D20894
2019-07-10 01:08:08 +00:00
Li-Wen Hsu
43527153dc Skip sys.netpfil.pf.names.names and sys.netpfil.pf.synproxy.synproxy
temporarily because kernel panics when flushing epair queue.

PR:		238870
Sponsored by:	The FreeBSD Foundation
2019-06-29 12:19:57 +00:00
Li-Wen Hsu
01e92e2977 Skip sys.netinet.socket_afinet.socket_afinet_bind_zero temporarily because it
doesn't work when mac_portacl(4) loaded

PR:		238781
Sponsored by:	The FreeBSD Foundation
2019-06-23 19:37:12 +00:00
Conrad Meyer
179f62805c random(4): Fortuna: allow increased concurrency
Add experimental feature to increase concurrency in Fortuna.  As this
diverges slightly from canonical Fortuna, and due to the security
sensitivity of random(4), it is off by default.  To enable it, set the
tunable kern.random.fortuna.concurrent_read="1".  The rest of this commit
message describes the behavior when enabled.

Readers continue to update shared Fortuna state under global mutex, as they
do in the status quo implementation of the algorithm, but shift the actual
PRF generation out from under the global lock.  This massively reduces the
CPU time readers spend holding the global lock, allowing for increased
concurrency on SMP systems and less bullying of the harvestq kthread.

It is somewhat of a deviation from FS&K.  I think the primary difference is
that the specific sequence of AES keys will differ if READ_RANDOM_UIO is
accessed concurrently (as the 2nd thread to take the mutex will no longer
receive a key derived from rekeying the first thread).  However, I believe
the goals of rekeying AES are maintained: trivially, we continue to rekey
every 1MB for the statistical property; and each consumer gets a
forward-secret, independent AES key for their PRF.

Since Chacha doesn't need to rekey for sequences of any length, this change
makes no difference to the sequence of Chacha keys and PRF generated when
Chacha is used in place of AES.

On a GENERIC 4-thread VM (so, INVARIANTS/WITNESS, numbers not necessarily
representative), 3x concurrent AES performance jumped from ~55 MiB/s per
thread to ~197 MB/s per thread.  Concurrent Chacha20 at 3 threads went from
roughly ~113 MB/s per thread to ~430 MB/s per thread.

Prior to this change, the system was extremely unresponsive with 3-4
concurrent random readers; each thread had high variance in latency and
throughput, depending on who got lucky and won the lock.  "rand_harvestq"
thread CPU use was high (double digits), seemingly due to spinning on the
global lock.

After the change, concurrent random readers and the system in general are
much more responsive, and rand_harvestq CPU use dropped to basically zero.

Tests are added to the devrandom suite to ensure the uint128_add64 primitive
utilized by unlocked read functions to specification.

Reviewed by:	markm
Approved by:	secteam(delphij)
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D20313
2019-06-17 20:29:13 +00:00
Xin LI
f89d207279 Separate kernel crc32() implementation to its own header (gsb_crc32.h) and
rename the source to gsb_crc32.c.

This is a prerequisite of unifying kernel zlib instances.

PR:		229763
Submitted by:	Yoshihiro Ota <ota at j.email.ne.jp>
Differential Revision:	https://reviews.freebsd.org/D20193
2019-06-17 19:49:08 +00:00
Conrad Meyer
d0d71d818c random(4): Generalize algorithm-independent APIs
At a basic level, remove assumptions about the underlying algorithm (such as
output block size and reseeding requirements) from the algorithm-independent
logic in randomdev.c.  Chacha20 does not have many of the restrictions that
AES-ICM does as a PRF (Pseudo-Random Function), because it has a cipher
block size of 512 bits.  The motivation is that by generalizing the API,
Chacha is not penalized by the limitations of AES.

In READ_RANDOM_UIO, first attempt to NOWAIT allocate a large enough buffer
for the entire user request, or the maximal input we'll accept between
signal checking, whichever is smaller.  The idea is that the implementation
of any randomdev algorithm is then free to divide up large requests in
whatever fashion it sees fit.

As part of this, two responsibilities from the "algorithm-generic" randomdev
code are pushed down into the Fortuna ra_read implementation (and any other
future or out-of-tree ra_read implementations):

  1. If an algorithm needs to rekey every N bytes, it is responsible for
  handling that in ra_read(). (I.e., Fortuna's 1MB rekey interval for AES
  block generation.)

  2. If an algorithm uses a block cipher that doesn't tolerate partial-block
  requests (again, e.g., AES), it is also responsible for handling that in
  ra_read().

Several APIs are changed from u_int buffer length to the more canonical
size_t.  Several APIs are changed from taking a blockcount to a bytecount,
to permit PRFs like Chacha20 to directly generate quantities of output that
are not multiples of RANDOM_BLOCKSIZE (AES block size).

The Fortuna algorithm is changed to NOT rekey every 1MiB when in Chacha20
mode (kern.random.use_chacha20_cipher="1").  This is explicitly supported by
the math in FS&K §9.4 (Ferguson, Schneier, and Kohno; "Cryptography
Engineering"), as well as by their conclusion: "If we had a block cipher
with a 256-bit [or greater] block size, then the collisions would not
have been an issue at all."

For now, continue to break up reads into PAGE_SIZE chunks, as they were
before.  So, no functional change, mostly.

Reviewed by:	markm
Approved by:	secteam(delphij)
Differential Revision:	https://reviews.freebsd.org/D20312
2019-06-17 15:09:12 +00:00
Conrad Meyer
403c041316 random(4): Add regression tests for uint128 implementation, Chacha CTR
Add some basic regression tests to verify behavior of both uint128
implementations at typical boundary conditions, to run on all architectures.

Test uint128 increment behavior of Chacha in keystream mode, as used by
'kern.random.use_chacha20_cipher=1' (r344913) to verify assumptions at edge
cases.  These assumptions are critical to the safety of using Chacha as a
PRF in Fortuna (as implemented).

(Chacha's use in arc4random is safe regardless of these tests, as it is
limited to far less than 4 billion blocks of output in that API.)

Reviewed by:	markm
Approved by:	secteam(gordon)
Differential Revision:	https://reviews.freebsd.org/D20392
2019-06-17 14:59:45 +00:00
Alan Somers
38edd595df Add test cases for epair
Implements the missing test cases for epair in a similar fashion to the
existing tests. Fixes shared abstractions to work with epair tests.

Submitted by:	Ryan Moeller <ryan@freqlabs.com>
Reviewed by:	asomers
MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D20498
2019-06-13 05:05:58 +00:00
Li-Wen Hsu
dff3f7f89b Remove tests for the deprecated algorithms in r348206
The tests are failing because the return value and output have changed, but
before test code structure adjusted, removing these test cases help people
be able to focus on more important cases.

Discussed with:	emaste
MFC with:	r348206
Sponsored by:	The FreeBSD Foundation
2019-05-31 04:29:29 +00:00
Bjoern A. Zeeb
aa32159686 Add very basic afinet socket tests which I started to write in order
to then try to reproduce a kernel panic, which turned out to be a
race condition and hard to test from here.

Commit the changes anywhere as the "bind zero" case was a surprise
to me and we should try to maintain this status.

Also it is easy examples someone can build upon.

With help from:	markj
Event:		Waterloo Hackathon 2019
2019-05-21 19:42:04 +00:00
Enji Cooper
f2a344455f Add my name to the copyright
I have contributed a number of changes to these tests over the past few
hundred revisions, and believe I deserve credit for the changes I have
made (plus, the copyright hadn't been updated since 2014).

MFC after:	1 week
2019-05-21 04:11:16 +00:00
Enji Cooper
ef02523dc8 Follow up to r348042: cast aad to a byte array
This is not completely necessary today, but this change is being made in a
conservative manner to avoid accidental breakage in the future, if this ever
was a unicode string.

PR:		237403
MFC after:	1 week
2019-05-21 04:03:22 +00:00
Enji Cooper
ffbc8cc0f2 Fix encoding issues with python 3
In python 3, the default encoding was switched from ascii character sets to
unicode character sets in order to support internationalization by default.
Some interfaces, like ioctls and packets, however, specify data in terms of
non-unicode encodings formats, either in host endian (`fcntl.ioctl`) or
network endian (`dpkt`) byte order/format.

This change alters assumptions made by previous code where it was all
data objects were assumed to be basestrings, when they should have been
treated as byte arrays. In order to achieve this the following are done:
* str objects with encodings needing to be encoded as ascii byte arrays are
  done so via `.encode("ascii")`. In order for this to work on python 3 in a
  type agnostic way (as it anecdotally varied depending on the caller), call
  `.encode("ascii")` only on str objects with python 3 to cast them to ascii
  byte arrays in a helper function name `str_to_ascii(..)`.
* `dpkt.Packet` objects needing to be passed in to `fcntl.ioctl(..)` are done
  so by casting them to byte arrays via `bytes()`, which calls
  `dpkt.Packet__str__` under the covers and does the necessary str to byte array
  conversion needed for the `dpkt` APIs and `struct` module.

In order to accomodate this change, apply the necessary typecasting for the
byte array literal in order to search `fop.name` for nul bytes.

This resolves all remaining python 2.x and python 3.x compatibility issues on
amd64. More work needs to be done for the tests to function with i386, in
general (this is a legacy issue).

PR:		237403
MFC after:	1 week
Tested with:	python 2.7.16 (amd64), python 3.6.8 (amd64)
2019-05-21 03:52:48 +00:00
Enji Cooper
f6d7fcda27 Remove spurious newline
Even though some python styles suggest there should be multiple newlines between
methods/classes, for consistency with the surrounding code, it's best to be
consistent by having merely one newline between each functional block.

MFC after:	1 week
2019-05-21 02:49:15 +00:00
Enji Cooper
a60d9a9892 Fix KAT(CCM)?Parser file descriptor leaks
Make `KAT(CCM)?Parser` into a context suite-capable object by implementing
`__enter__` and `__exit__` methods which manage opening up the file descriptors
and closing them on context exit. This implementation was decided over adding
destructor logic to a `__del__` method, as there are a number of issues around
object lifetimes when dealing with threading cleanup, atexit handlers, and a
number of other less obvious edgecases. Plus, the architected solution is more
pythonic and clean.

Complete the iterator implementation by implementing a `__next__` method for
both classes which handles iterating over the data using a generator pattern,
and by changing `__iter__` to return the object instead of the data which it
would iterate over. Alias the `__next__` method to `next` when working with
python 2.x in order to maintain functional compatibility between the two major
versions.

As part of this work and to ensure readability, push the initialization of the
parser objects up one layer and pass it down to a helper function. This could
have been done via a decorator, but I was trying to keep it simple for other
developers to make it easier to modify in the future.

This fixes ResourceWarnings with python 3.

PR:		237403
MFC after:	1 week
Tested with:	python 2.7.16 (amd64), python 3.6.8 (amd64)
2019-05-21 02:30:43 +00:00
Enji Cooper
8c02634818 Squash deprecation warning related to array.array(..).tostring()
In version 3.2+, `array.array(..).tostring()` was renamed to
`array.array(..).tobytes()`. Conditionally call `array.array(..).tobytes()` if
the python version is 3.2+.

PR:		237403
MFC after:	1 week
2019-05-21 02:13:46 +00:00
Enji Cooper
e8b4bbdfae Followup to r347996
Replace uses of `foo.encode("hex")` with `binascii.hexlify(foo)` for forwards
compatibility between python 2.x and python 3.

PR:		237403
MFC after:	1 week
2019-05-21 00:30:29 +00:00
Enji Cooper
1466140817 Allow the end-user to pass along arguments to cryptotest.py via $CRYPTOTEST_ARGS
This allows someone to use `-v` to dump out standard output.
2019-05-20 22:32:26 +00:00
Edward Tomasz Napierala
849f6862ff Improve tree(3) tests by using ATF_REQUIRE where applicable.
MFC after:	2 weeks
Sponsored by:	Klara Inc.
2019-05-20 18:35:23 +00:00
Enji Cooper
d99c2cecc8 Replace uses of foo.(de|en)code('hex') with binascii.(un)?hexlify(foo)
Python 3 no longer doesn't support encoding/decoding hexadecimal numbers using
the `str.format` method. The backwards compatible new method (using the
binascii module/methods) is a comparable means of converting to/from
hexadecimal format.

In short, the functional change is the following:
* `foo.decode('hex')` -> `binascii.unhexlify(foo)`
* `foo.encode('hex')` -> `binascii.hexlify(foo)`

While here, move the dpkt import in `cryptodev.py` down per PEP8, so it comes
after the standard library provided imports.

PR:		237403
MFC after:	1 week
2019-05-20 16:38:12 +00:00
Kristof Provost
b685454a11 ipsec tests: Skip if ipsec.ko is not loaded
As of r347410 IPSec is no longer built into GENERIC. The ipsec.ko module must
be loaded before we can execute the IPSec tests.

Check this, and skip the tests if IPSec is not available.
2019-05-16 02:18:57 +00:00
Edward Tomasz Napierala
16e55b9e0e Try to unbreak the build after r347425.
MFC after:	2 weeks
2019-05-10 08:16:29 +00:00
Edward Tomasz Napierala
36a040183e Add simple regression tests for tree(3). Those are ATF-ified versions
of OpenBSD's regress/sys/sys/tree/.

Reviewed by:	ngie
MFC after:	2 weeks
Sponsored by:	Klara Inc.
Differential Revision:	https://reviews.freebsd.org/D20186
2019-05-10 07:46:14 +00:00
Enji Cooper
16f35864df Refactor tests/sys/opencrypto/runtests
* Convert from plain to TAP for slightly improved introspection when skipping
  the tests due to requirements not being met.
* Test for the net/py-dpkt (origin) package being required when running the
  tests, instead of relying on a copy of the dpkt.py module from 2014. This
  enables the tests to work with py3. Subsequently, remove
  `tests/sys/opencrypto/dpkt.py(c)?` via `make delete-old`.
* Parameterize out `python2` as `$PYTHON`.

PR:		237403
MFC after:	1 week
2019-05-10 00:03:32 +00:00
Mark Johnston
8beadca53c Add a few regression tests for mlock(2).
These are intended to exercise some rarely executed code paths.

MFC after:	2 weeks
2019-05-01 15:28:23 +00:00
Enji Cooper
351a56b116 Use range instead of xrange
`xrange` is a pre-python 2.x compatible idiom. Use `range` instead. The values
being iterated over are sufficiently small that using range on python 2.x won't
be a noticeable issue.

MFC after:	2 months
2019-04-24 05:52:24 +00:00
Enji Cooper
2a96ae15f0 Fix typo: Plen should be plen
MFC after:	1 month
MFC with:	r346617
Reported by:	pylint -E
2019-04-24 05:49:48 +00:00
Enji Cooper
56bf253633 Don't leak fd when manipulating the device via _getdev()
Close the file descriptor when done calling ioctl with a try-finally block so
it doesn't get leaked.

MFC after:	2 months
2019-04-24 05:47:09 +00:00
Enji Cooper
b106e0fccc Chase PEP-3110
Replace `except Environment, e:` with `except Environment as e` for
compatibility between python 2.x and python 3.x.

While here, fix a bad indentation change from r346620 by reindenting the code
properly.

MFC after:	2 months
2019-04-24 04:50:03 +00:00
Enji Cooper
ac65c82761 Reapply whitespace style changes from r346443 after recent changes to tests/sys/opencrypto
From r346443:
"""
Replace hard tabs with four-character indentations, per PEP8.

This is being done to separate stylistic changes from the tests from functional
ones, as I accidentally introduced a bug to the tests when I used four-space
indentation locally.

No functional change.
"""

MFC after:	2 months
Discussed with:	jhb
2019-04-24 04:40:24 +00:00
John Baldwin
151f0ca897 Test the AES-CCM test vectors from the NIST Known Answer Tests.
The CCM test vectors use a slightly different file format in that
there are global key-value pairs as well as section key-value pairs
that need to be used in each test.  In addition, the sections can set
multiple key-value pairs in the section name.  The CCM KAT parser
class is an iterator that returns a dictionary once per test where the
dictionary contains all of the relevant key-value pairs for a given
test (global, section name, section, test-specific).

Note that all of the CCM decrypt tests use nonce and tag lengths that
are not supported by OCF (OCF only supports a 12 byte nonce and 16
byte tag), so none of the decryption vectors are actually tested.

Reviewed by:	ngie
MFC after:	1 month
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D19978
2019-04-24 00:23:06 +00:00
John Baldwin
de0f7dca5e Run the plain SHA digest tests from NIST.
Pass in an explicit digest length to the Crypto constructor since it
was assuming only sessions with a MAC key would have a MAC.  Passing
an explicit size allows us to test the full digest in HMAC tests as
well.

Reviewed by:	cem
MFC after:	1 month
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D19884
2019-04-24 00:16:39 +00:00
John Baldwin
c091d0d95d Use more descriptive algorithm names in skip messages.
Reviewed by:	cem, ngie
MFC after:	1 month
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D19977
2019-04-24 00:14:37 +00:00
John Baldwin
aeb5c8e609 Skip tests with missing test vectors instead of failing.
This copes more gracefully when older version of the nist-kat package
are intalled that don't have newer test vectors such as CCM or plain
SHA.

If the nist-kat package is not installed at all, this still fails with
an error.

Reviewed by:	cem
MFC after:	1 month
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D20034
2019-04-24 00:10:21 +00:00
Olivier Cochard
9583ab8aa3 Skip test component_selection:run_latest_genid if gmirror/gnop GEOM classes
aren't available

PR:		237051
Reviewed by:	asomers, imp, ngie, emaste (IRC)
Approved by:	ngie
MFC after:	 1 month
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D19958
2019-04-23 21:07:47 +00:00
Enji Cooper
616f60e87e Fix sys.kern.coredump_phnum_test.coredump_phnum on i386
The zero-padding when printing out the Size field is on 32-bit architectures is
5, not 15. Adjust the regular expression to work with both the 32-bit and
64-bit case.

MFC after:	1 week
Reviewed by:	lwhsu, markj
Approved by:	emaste (mentor, implicit)
Differential Revision: https://reviews.freebsd.org/D20005
2019-04-22 11:21:20 +00:00
Enji Cooper
03accca747 Revert r346443
My wide sweeping stylistic change (while well intended) is impeding others from
working on `tests/sys/opencrypto`.

The plan is to revert the change in ^/head, then reintroduce the changes after
the other changes get merged into ^/head .

Approved by:	emaste (mentor; implicit)
Requested by:	jhb
MFC after:	2 months
2019-04-20 16:37:28 +00:00
Enji Cooper
7bd1cac6c5 tests/sys/opencrypto: fix whitespace per PEP8
Replace hard tabs with four-character indentations, per PEP8.

This is being done to separate stylistic changes from the tests from functional
ones, as I accidentally introduced a bug to the tests when I used four-space
indentation locally.

No functional change.

MFC after:	2 months
Approved by:	emaste (mentor: implicit blanket approval for trivial fixes)
2019-04-20 15:43:28 +00:00