Commit Graph

1006 Commits

Author SHA1 Message Date
Alan Somers
7d4670b647 ZFS: the hotspare_add_004_neg test needs at least two disks
MFC after:	2 weeks
Sponsored by:	Axcient
2019-10-05 01:34:35 +00:00
Kyle Evans
5a391b572b shm_open2(2): completely unbreak
kern_shm_open2(), since conception, completely fails to pass the mode along
to kern_shm_open(). This breaks most uses of it.

Add tests alongside this that actually check the mode of the returned
files.

PR:		240934 [pulseaudio breakage]
Reported by:	ler, Andrew Gierth [postgres breakage]
Diagnosed by:	Andrew Gierth (great catch)
Tested by:	ler, tmunro
Pointy hat to:	kevans
2019-10-02 02:37:34 +00:00
Eric van Gyzen
2b2ad2d6c6 Fix coredump_phnum_test when kern.compress_user_cores != 0
If `kern.compress_user_cores` is non-zero, decompress the core file.

Use `sysctl -f` to restore previous values.

Don't bother restoring `ulimit -c`, since that's a per-process value.

Check more commands with `atf_check`.

Reviewed by:	olivier ngie
MFC after:	1 week
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D21820
2019-09-30 14:05:44 +00:00
Jilles Tjoelker
4f77551258 Adjust tests after page fault changes in r352807
Commit r352807 fixed various signal numbers and codes from page faults;
adjust the tests so they expect the fixes to be present.

PR:		211924
2019-09-29 15:17:58 +00:00
Kyle Evans
e12ff89136 Further normalize copyright notices
- s/C/c/ where I've been inconsistent about it
- +SPDX tags
- Remove "All rights reserved" where possible

Requested by:	rgrimes (all rights reserved)
2019-09-26 16:19:22 +00:00
David Bright
9afb12bab4 Add an shm_rename syscall
Add an atomic shm rename operation, similar in spirit to a file
rename. Atomically unlink an shm from a source path and link it to a
destination path. If an existing shm is linked at the destination
path, unlink it as part of the same atomic operation. The caller needs
the same permissions as shm_unlink to the shm being renamed, and the
same permissions for the shm at the destination which is being
unlinked, if it exists. If those fail, EACCES is returned, as with the
other shm_* syscalls.

truss support is included; audit support will come later.

This commit includes only the implementation; the sysent-generated
bits will come in a follow-on commit.

Submitted by:	Matthew Bryan <matthew.bryan@isilon.com>
Reviewed by:	jilles (earlier revision)
Reviewed by:	brueffer (manpages, earlier revision)
Relnotes:	yes
Sponsored by:	Dell EMC Isilon
Differential Revision:	https://reviews.freebsd.org/D21423
2019-09-26 15:32:28 +00:00
Kyle Evans
a631497fca Add SPDX tags to recently added files
Reported by:	Pawel Biernacki
2019-09-25 22:53:30 +00:00
Kyle Evans
3e25d1fb61 Add linux-compatible memfd_create
memfd_create is effectively a SHM_ANON shm_open(2) mapping with optional
CLOEXEC and file sealing support. This is used by some mesa parts, some
linux libs, and qemu can also take advantage of it and uses the sealing to
prevent resizing the region.

This reimplements shm_open in terms of shm_open2(2) at the same time.

shm_open(2) will be moved to COMPAT12 shortly.

Reviewed by:	markj, kib
Differential Revision:	https://reviews.freebsd.org/D21393
2019-09-25 18:03:18 +00:00
Olivier Cochard
13d9bd2692 Fix coredump_phnum_test in case of kern.compress_user_cores=1
PR:		240783
Approved by:	ngie, lwhsu
MFC after:	1 month
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D21776
2019-09-24 16:45:34 +00:00
Jilles Tjoelker
416e2de337 Add some tests for page fault signals and codes
It is useful to have some tests for page fault signals.

More tests would be useful but creating the conditions (such as various
kinds of running out of memory and I/O errors) is more complicated.

The tests page_fault_signal__bus_objerr_1 and
page_fault_signal__bus_objerr_2 depend on https://reviews.freebsd.org/D21566
before they can pass.

PR:		211924
Reviewed by:	kib
Differential Revision:	https://reviews.freebsd.org/D21624
2019-09-18 21:00:32 +00:00
Li-Wen Hsu
ebcb81079e Temporarily skip sys.netpfil.common.tos.pf_tos on i386 CI as it always fails
PR:		240086
Sponsored by:	The FreeBSD Foundation
2019-09-17 22:09:14 +00:00
Li-Wen Hsu
ae92090ae0 Temporarily skip sys.netpfil.common.forward.pf_v4 on i386 CI as it always fails
PR:		240085
Sponsored by:	The FreeBSD Foundation
2019-09-17 22:08:16 +00:00
Alan Somers
9c9634d1db fusefs: initialize C++ classes the Coverity way
Coverity complained that I wasn't initializing some class members until the
SetUp method.  Do it in the constructor instead.

Reported by:	Coverity
Coverity CIDs:	1404352, 1404378
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
2019-09-16 15:56:21 +00:00
Alan Somers
4ca1c0b7f7 fusefs: fix some minor Coverity CIDs in the tests
Where open(2) is expected to fail, the tests should assert or expect that
its return value is -1.  These tests all accepted too much but happened to
pass anyway.

Reported by:	Coverity
Coverity CID:	1404512, 1404378, 1404504, 1404483
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
2019-09-16 15:44:59 +00:00
Edward Tomasz Napierala
fad4b12b90 Introduce arb(3), the Array-based Red-Black Tree macros: similar
to the traditional tree(3) RB trees, but using an array (preallocated,
linear chunk of memory) to store the tree.

This avoids allocation overhead, improves memory locality,
and makes it trivially easy to share/transfer/copy the entire tree
without the need for marshalling.  The downside is that the size
is fixed at initialization time; there is no mechanism to resize
it.

This is one of the dependencies for the new stats(3) framework
(https://reviews.freebsd.org/D20477).

Reviewed by:	bcr (man pages), markj
Discussed with:	cem
MFC after:	2 weeks
Sponsored by:	Klara Inc, Netflix
Obtained from:	Netflix
Differential Revision:	https://reviews.freebsd.org/D20324
2019-09-14 19:23:46 +00:00
Alan Somers
6c0c362075 fusefs: Fix iosize for FUSE_WRITE in 7.8 compat mode
When communicating with a FUSE server that implements version 7.8 (or older)
of the FUSE protocol, the FUSE_WRITE request structure is 16 bytes shorter
than normal. The protocol version check wasn't applied universally, leading
to an extra 16 bytes being sent to such servers. The extra bytes were
allocated and bzero()d, so there was no information disclosure.

Reviewed by:	emaste
MFC after:	3 days
MFC-With:	r350665
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D21557
2019-09-11 19:29:40 +00:00
Li-Wen Hsu
31e3dc2da3 Temporarily skip flakey test case sys.kern.ptrace_test.ptrace__getppid
PR:		240510
Sponsored by:	The FreeBSD Foundation
2019-09-11 16:24:03 +00:00
Alan Somers
dd21a9163c fusefs: suppress some Coverity resource leak CIDs in the tests
The fusefs tests deliberately leak file descriptors.  To do otherwise would
add extra complications to the tests' mock FUSE server.  This annotation
should hopefully convince Coverity to shut up about the leaks.

Reviewed by:	uqs
MFC after:	4 days
Sponsored by:	The FreeBSD Foundation
2019-09-07 19:25:45 +00:00
Alan Somers
8e7657374a fusefs: coverity cleanup in the tests
Address the following defects reported by Coverity:

* Structurally dead code (CID 1404366): set m_quit before FAIL, not after

* Unchecked return value of sysctlbyname (CID 1404321)

* Unchecked return value of stat(2) (CID 1404471)

* Unchecked return value of open(2) (CID 1404402, 1404529)

* Unchecked return value of dup(2) (CID 1404478)

* Buffer overflows. These are all false positives caused by the fact that
  Coverity thinks I'm using a buffer to store strings, when in fact I'm
  really just using it to store a byte array that happens to be initialized
  with a string. I'm changing the type from char to uint8_t in the hopes
  that it will placate Coverity. (CID 1404338, 1404350, 1404367, 1404376,
  1404379, 1404381, 1404388, 1404403, 1404425, 1404433, 1404434, 1404474,
  1404480, 1404484, 1404503, 1404505)

* False positive file descriptor leak. I'm going to try to fix this with
  Coverity modeling, but I'll also change an EXPECT to ASSERT so we don't
  perform meaningless assertions after the failure. (CID 1404320, 1404324,
  1404440, 1404445).

* Unannotated file descriptor leak. This will be followed up by a Coverity
  modeling change. (CID 1404326, 1404334, 1404336, 1404357, 1404361,
  1404372, 1404391, 1404395, 1404409, 1404430, 1404448, 1404451, 1404455,
  1404457, 1404458, 1404460)

* Uninitialized variables in C++ constructors (CID 1404327, 1404346). In the
  case of m_maxphys, this actually led to part of the FUSE_INIT's response
  being set to stack garbage during the WriteCluster::clustering test.

* Uninitialized sun_len field in struct sockaddr_un (CID 1404330, 1404371,
  1404429).

Reported by:	Coverity
Reviewed by:	emaste
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D21457
2019-09-06 19:50:45 +00:00
Kristof Provost
3bc534ea8e Set required program for all acl tests
r339782 re-enabled acl test 00 and 02, which were disabled in r336617
due to PR 229930.
When the tests were disabled the code to set their required programs was
disabled as well, but this was not reinstated when r339782 re-enabled
them.
Do so now.

Sponsored by:	Axiado
2019-09-05 18:54:46 +00:00
Emmanuel Vadot
ea8a4d593b pkgbase: Put the sys/common test into the tests package
Every other test is there so do the same for those.

Reviewed by:	bapt, gjb
Differential Revision:	https://reviews.freebsd.org/D21504
2019-09-05 14:14:07 +00:00
Li-Wen Hsu
7fb6c523bc Temporarily skip sys.sys.qmath_test.qdivq_s64q in CI because it is unstable
PR:		240219
Discussed with:	trasz
Sponsored by:	The FreeBSD Foundation
2019-09-03 10:49:13 +00:00
Edward Tomasz Napierala
3dd61c212c Fix qmath(3) build problems with GCC 8.
Discussed with:	asomers
Sponsored by:	Klara Systems
Differential Revision:	https://reviews.freebsd.org/D21442
2019-08-29 07:39:31 +00:00
Alan Somers
5e63333052 fusefs: Fix some bugs regarding the size of the LISTXATTR list
* A small error in r338152 let to the returned size always being exactly
  eight bytes too large.

* The FUSE_LISTXATTR operation works like Linux's listxattr(2): if the
  caller does not provide enough space, then the server should return ERANGE
  rather than return a truncated list.  That's true even though in FUSE's
  case the kernel doesn't provide space to the client at all; it simply
  requests a maximum size for the list.  We previously weren't handling the
  case where the server returns ERANGE even though the kernel requested as
  much size as the server had told us it needs; that can happen due to a
  race.

* We also need to ensure that a pathological server that always returns
  ERANGE no matter what size we request in FUSE_LISTXATTR won't cause an
  infinite loop in the kernel.  As of this commit, it will instead cause an
  infinite loop that exits and enters the kernel on each iteration, allowing
  signals to be processed.

Reviewed by:	cem
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D21287
2019-08-28 04:19:37 +00:00
Edward Tomasz Napierala
912b6fa37e Fix build on 32 bit archs. 2019-08-27 13:55:45 +00:00
Edward Tomasz Napierala
5a38af1fb8 Introduce <sys/qmath.h>, a fixed-point math library from Netflix.
This makes it possible to perform mathematical operations
on
fractional values without using floating point. It operates on Q
numbers, which are integer-sized, opaque structures initialized
to hold a chosen number of integer and fractional
bits.


For a general description of the Q number system, see the "Fixed Point
Representation & Fractional Math" whitepaper[1]; for the actual
API see the qmath(3) man page.

This is one of dependencies for the upcoming stats(3) framework[2]
that will be applied to the TCP stack in a later commit.

1. https://www.superkits.net/whitepapers/Fixed%20Point%20Representation%20&%20Fractional%20Math.pdf
2. https://reviews.freebsd.org/D20477

Reviewed by:	bcr (man pages, earlier version), sef (earlier version)
Discussed with:	cem, dteske, imp, lstewart
Sponsored By:	Klara Inc, Netflix
Obtained from:	Netflix
Differential Revision:	https://reviews.freebsd.org/D20116
2019-08-27 11:46:22 +00:00
Kyle Evans
95aa96f3a8 tests: shm_open(2): Verify FD_CLOEXEC
Motivated by the fact that I'm messing around near the implementation and
wanting to ensure this doesn't get messed up in the process.

MFC after:	1 week
2019-08-25 02:38:45 +00:00
Li-Wen Hsu
74fa3a52ce Fix failure test cases after r351423 due to ping6(8) options changed
Failure test cases:
    sys.netpfil.common.pass_block.pf_v6
    sys.netpfil.pf.pass_block.noalias
    sys.netpfil.pf.pass_block.v6

Sponsored by:	The FreeBSD Foundation
2019-08-24 01:28:39 +00:00
Kristof Provost
39cae0d5ed netpfil tests: Add forward test for the three firewalls
Submitted by:	Ahsan Barkati
Reviewed by:	kp
Sponsored by:	Google, Inc. (GSoC 2019)
Differential Revision:	https://reviews.freebsd.org/D21321
2019-08-23 12:11:46 +00:00
Kyle Evans
e21f96a811 mips: hide regnum definitions behind _KERNEL/_WANT_MIPS_REGNUM
machine/regnum.h ends up being included by sys/procfs.h and sys/ptrace.h via
machine/reg.h. Many of the regnum definitions are too short and too generic
to be exposing to any userland application including one of these two
headers. Moreover, these actively cause build failures in googletest
(template <typename T1 ...> expanding to template <typename 9 ...>).

Hide the definitions behind _KERNEL or _WANT_MIPS_REGNUM, and patch all of
the userland consumers to define as needed.

Discussed with:	imp, jhb
Reviewed by:	imp, jhb
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D21330
2019-08-22 21:43:21 +00:00
Kyle Evans
92a8983bab Fix the build with WITHOUT_GOOGLETEST
Attempting to build the fusefs tests WITHOUT_GOOGLETEST will result in an
error if the host system or sysroot doesn't already have googletest headers
in /usr/include/private (e.g. host built/installed WITHOUT_GOOGLETEST, clean
cross-buildworld WITHOUT_GOOGLETEST).

Reviewed by:	asomers
Differential Revision:	https://reviews.freebsd.org/D21367
2019-08-22 19:10:31 +00:00
Li-Wen Hsu
cdac716946 Fix path issues after r351212
This fixes sys.netpfil.pf.forward.v4 and sys.netpfil.pf.icmp.cve_2019_5598
failures in CI system.

Sponsored by:	The FreeBSD Foundation
2019-08-22 12:08:35 +00:00
Kristof Provost
3e1ebe7fb8 netpfil tests: Add too many fragments test for pf, ipfw and ipf
Add test for checking that the packets are dropped if it is fragmented into
more than the defined value.

Submitted by:	Ahsan Barkati
Reviewed by:	kp
Sponsored by:	Google, Inc. (GSoC 2019)
Differential Revision:	https://reviews.freebsd.org/D21307
2019-08-20 14:46:12 +00:00
Kristof Provost
df3d236d96 netpfil tests: Add the set ToS test for ipfw and pf
This test tests the following:

 - The firewall is able to set the tos bits
 - The firewall is able to set the DSCP bits when EN bits is already set and
   the EN bits remains unchanged.
 - The firewall is able to drop the packets based on ToS value

Submitted by:	Ahsan Barkati
Reviewed by:	kp
Sponsored by:	Google, Inc. (GSoC 2019)
Differential Revision:	https://reviews.freebsd.org/D21305
2019-08-20 14:31:22 +00:00
Kristof Provost
9531253098 netpfil tests: Move pft_ping.py and sniffer.py to the common test directory
The pft_ping.py and sniffer.py tool is moved from tests/sys/netpfil/pf to
tests/sys/netpfil/common directory because these tools are to be used in
common for all the firewalls.

Submitted by:	Ahsan Barkati
Reviewed by:	kp, thj
Sponsored by:	Google, Inc. (GSoC 2019)
Differential Revision:	https://reviews.freebsd.org/D21276
2019-08-19 10:48:27 +00:00
Konstantin Belousov
de567a4bef sys.kern.pdeathsig.signal_delivered_ptrace: fix startup.
Inform D that C executed procctl(PROC_PDEATHSIG_CTL).  Otherwise D
might allow B to exit before C is set up to receive a signal on the
parent exit.  In this case, C waits forever for the signal and test
hangs.

PR:	237657
Reported and tested by:	lwhsu
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2019-08-19 09:33:09 +00:00
Konstantin Belousov
39ba52c0b7 sys.kern.pdeathsig.signal_delivered_ptrace: fix debugger detach
ptrace(PT_DETACH) requires stopped debuggee, otherwise it fails.  When
the call fails, the C process is left as debuggee of the process D,
and might be killed too early if process D exits occurs fast enough.

Since pipes are not closed in the forked children, this resulted in
the test hanging, since no write occured from C to wake A.

PR:	237657
Reported and tested by:	lwhsu
Sponsored by:	The FreeBSD Foundation
MFC after:	1 week
2019-08-19 09:29:27 +00:00
Tom Jones
0d9da68f01 Add basic NAT test for pf, ipf and ipfw
Add common firewall NAT tests for pf, ipf and ipfw (using both in-kernel and
userspace NAT).

Submitted by:   Ahsan Barkati
Sponsored by:   Google, Inc. (GSoC 2019)
Reviewed by:    kp
Approved by:    bz (mentor)
MFC after:      2 weeks
Differential Revision: https://reviews.freebsd.org/D21199
2019-08-17 06:44:11 +00:00
Alan Somers
3a79e8e772 fusefs: don't send the namespace during listextattr
The FUSE_LISTXATTR operation always returns the full list of a file's
extended attributes, in all namespaces. There's no way to filter the list
server-side. However, currently FreeBSD's fusefs driver sends a namespace
string with the FUSE_LISTXATTR request. That behavior was probably copied
from fuse_vnop_getextattr, which has an attribute name argument. It's
been there ever since extended attribute support was added in r324620. This
commit removes it.

Reviewed by:	cem
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D21280
2019-08-16 05:06:54 +00:00
Mateusz Piotrowski
03d8a4b7d3 pf tests: Fix accidental duplication of content
Some files got their contented duplicated in r345409. Some mistakes where
fixed in r345430. The only file that was left with a duplicated content was
CVE-2019-5598.py.

Reviewed by:	kp
Approved by:	src (kp)
Differential Revision:	https://reviews.freebsd.org/D21267
2019-08-15 12:00:59 +00:00
Alan Somers
38a3e0bd6b fusefs: fix conditional from r351061
The entirety of r351061 was a copy/paste error.  I'm sorry I've been
comitting so hastily.

Reported by:	rpokala
Reviewed by:	rpokala
MFC after:	2 weeks
MFC-With:	351061
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D21265
2019-08-15 04:47:42 +00:00
Alan Somers
f6b344e560 fusefs: fix the 32-bit build after 351042
Reported by:	jhb
MFC after:	2 weeks
MFC-With:	351042
Sponsored by:	The FreeBSD Foundation
2019-08-15 00:23:03 +00:00
Alan Somers
bf50749773 fusefs: Fix the size of fuse_getattr_in
In FUSE protocol 7.9, the size of the FUSE_GETATTR request has increased.
However, the fusefs driver is currently not sending the additional fields.
In our implementation, the additional fields are always zero, so I there
haven't been any test failures until now.  But fusefs-lkl requires the
request's length to be correct.

Fix this bug, and also enhance the test suite to catch similar bugs.

PR:		239830
MFC after:	2 weeks
MFC-With:	350665
Sponsored by:	The FreeBSD Foundation
2019-08-14 20:45:00 +00:00
Alan Somers
331884f291 fusefs: fix intermittency in the default_permissions.Unlink.ok test
The test needs to expect a FUSE_FORGET operation. Most of the time the test
would pass anyway, because by chance FUSE_FORGET would arrive after the
unmount.

MFC after:	2 weeks
MFC-With:	350665
Sponsored by:	The FreeBSD Foundation
2019-08-14 18:04:04 +00:00
Alan Somers
dbee856aff fusefs: skip some tests when unsafe aio is disabled
MFC after:      15 days
MFC-With:       r350665
Sponsored by:   The FreeBSD Foundation
2019-08-13 15:52:28 +00:00
Alan Somers
1fa8ebfbbb fusefs: add SVN Keywords to the test files
Reported by:	SVN pre-commit hooks
MFC after:	15 days
MFC-With:	r350665
Sponsored by:	The FreeBSD Foundation
2019-08-13 15:49:40 +00:00
Alan Somers
c2265ae7a8 fusefs: skip some tests when unsafe aio is disabled
MFC after:	16 days
MFC-With:	r350665
Sponsored by:	The FreeBSD Foundation
2019-08-12 20:00:21 +00:00
Enji Cooper
84a457c6c0 tests/sys/opencrypto: enable armv8crypto on aarch64
This change makes required modifications in runtests to also only require the
aesni module on Intel (i386/amd64) platforms, as it is an Intel specific
module.

MFC after:	1 month
MFC to:		^/stable/12 (support not present on ^/stable/11)
Submitted by:	Greg V <greg@unrelenting.technology>
Differential Revision:	https://reviews.freebsd.org/D21018
2019-08-10 15:53:42 +00:00
Brooks Davis
4a045a66fd Don't add -Wno-class-memaccess with older gcc.
This is a gcc 8.0+ warning which needed to be silenced on for the riscv
build.  amd64-xtoolchain-gcc still uses gcc 6.4.0 and does not understand
this flag.

Reviewed by:	asomers
Feedback from:	imp
Differential Revision:	https://reviews.freebsd.org/D21195
2019-08-09 23:50:57 +00:00
Li-Wen Hsu
84922a5d4c Get configuration variable with default value for not breaking default setting
Reported by:	markj
Sponsored by:	The FreeBSD Foundation
2019-08-07 21:02:23 +00:00
Alan Somers
0b4275accb fusefs: merge from projects/fuse2
This commit imports the new fusefs driver. It raises the protocol level
from 7.8 to 7.23, fixes many bugs, adds a test suite for the driver, and
adds many new features. New features include:

* Optional kernel-side permissions checks (-o default_permissions)
* Implement VOP_MKNOD, VOP_BMAP, and VOP_ADVLOCK
* Allow interrupting FUSE operations
* Support named pipes and unix-domain sockets in fusefs file systems
* Forward UTIME_NOW during utimensat(2) to the daemon
* kqueue support for /dev/fuse
* Allow updating mounts with "mount -u"
* Allow exporting fusefs file systems over NFS
* Server-initiated invalidation of the name cache or data cache
* Respect RLIMIT_FSIZE
* Try to support servers as old as protocol 7.4

Performance enhancements include:

* Implement FUSE's FOPEN_KEEP_CACHE and FUSE_ASYNC_READ flags
* Cache file attributes
* Cache lookup entries, both positive and negative
* Server-selectable cache modes: writethrough, writeback, or uncached
* Write clustering
* Readahead
* Use counter(9) for statistical reporting

PR:		199934 216391 233783 234581 235773 235774 235775
PR:		236226 236231 236236 236291 236329 236381 236405
PR:		236327 236466 236472 236473 236474 236530 236557
PR:		236560 236844 237052 237181 237588 238565
Reviewed by:	bcr (man pages)
Reviewed by:	cem, ngie, rpokala, glebius, kib, bde, emaste (post-commit
		review on project branch)
MFC after:	3 weeks
Relnotes:	yes
Sponsored by:	The FreeBSD Foundation
Pull Request:	https://reviews.freebsd.org/D21110
2019-08-07 00:38:26 +00:00
Jilles Tjoelker
cb71f1fa01 Add a test for kill() on a zombie 2019-08-06 21:17:22 +00:00
Mariusz Zaborski
fd631bcd95 procdesc: fix reparenting when the debugger is attached
The process is reparented to the debugger while it is attached.
  B          B
 /   ---->   |
A          A D

Every time when the process is reparented, it is added to the orphan list
of the previous parent:

A->orphan = B
D->orphan = NULL

When the A process will close the process descriptor to the B process,
the B process will be reparented to the init process.
  B            B - init
  |   ---->
A D          A   D

A->orphan = B
D->orphan = B

In this scenario, the B process is in the orphan list of A and D.

When the last process descriptor is closed instead of reparenting
it to the reaper let it stay with the debugger process and set
our previews parent to the reaper.

Add test case for this situation.
Notice that without this patch the kernel will crash with this test case:
panic: orphan 0xfffff8000e990530 of 0xfffff8000e990000 has unexpected oppid 1

Reviewed by:	markj, kib
MFC after:	1 month
Differential Revision:	https://reviews.freebsd.org/D20361
2019-08-05 20:15:46 +00:00
Tom Jones
f97a8a3615 Add common firewall test suite
Add a common test suite for the firewalls included in the base system. The test
suite allows common test infrastructure to test pf, ipfw and ipf firewalls from
test files containing the setup for all three firewalls.

Add the pass block test for pf, ipfw and ipf. The pass block test checks the
allow/deny functionality of the firewalls tested.

Submitted by:   Ahsan Barkati
Sponsored by:   Google, Inc. (GSoC 2019)
Reviewed by:    kp
Approved by:    bz (co-mentor)
MFC after:      2 weeks
Differential Revision: https://reviews.freebsd.org/D21065
2019-08-05 11:47:34 +00:00
Li-Wen Hsu
9777e3544a Only skip test cases sometimes failing in CI when they are running in CI
Suggested by:	jhb
Sponsored by:	The FreeBSD Foundation
2019-08-01 18:19:16 +00:00
Alan Somers
9f13765e42 fusefs: fix building tests with GCC 8
GCC 8 objected to including C++-only flags in CWARNFLAGS

Sponsored by:	The FreeBSD Foundation
2019-07-30 19:47:45 +00:00
Alan Somers
f0c07f0ce8 fusefs: nul-terminate some strings in the readdir test
Reported by:	GCC 8
Sponsored by:	The FreeBSD Foundation
2019-07-30 17:31:09 +00:00
Alan Somers
669a092af1 fusefs: fix panic when writing with O_DIRECT and using writeback cache
When a fusefs file system is mounted using the writeback cache, the cache
may still be bypassed by opening a file with O_DIRECT.  When writing with
O_DIRECT, the cache must be invalidated for the affected portion of the
file.  Fix some panics caused by inadvertently invalidating too much.

Sponsored by:	The FreeBSD Foundation
2019-07-28 15:17:32 +00:00
Alan Somers
a63915c2d7 MFHead @r350386
Sponsored by:	The FreeBSD Foundation
2019-07-28 04:02:22 +00:00
Li-Wen Hsu
1ab93d1f23 Temporarily skip flakey test case
sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger

PR:		239425
Sponsored by:	The FreeBSD Foundation
2019-07-24 17:41:40 +00:00
Li-Wen Hsu
c2dc497a38 Temporarily skip flakey test case
sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger

PR:		239399
Sponsored by:	The FreeBSD Foundation
2019-07-23 09:39:27 +00:00
Li-Wen Hsu
ea24861d5e Temporarily skip flakey test case
sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger

PR:		239397
Sponsored by:	The FreeBSD Foundation
2019-07-23 09:19:58 +00:00
Li-Wen Hsu
7d1f74716c Temporarily skip flakey test case
sys.kern.ptrace_test.ptrace__PT_KILL_competing_stop

PR:		220841
Sponsored by:	The FreeBSD Foundation
2019-07-23 07:56:42 +00:00
Li-Wen Hsu
4f6d74c9c4 Temporarily skip sys.netpfil.pf.forward.{v4,v6} and sys.netpfil.pf.set_tos.v4
on i386 as they are flakey on it

PR:		239380
Sponsored by:	The FreeBSD Foundation
2019-07-22 18:54:26 +00:00
Li-Wen Hsu
63b0609c12 Fix URL.
Sponsored by:	The FreeBSD Foundation
2019-07-22 18:43:46 +00:00
Li-Wen Hsu
37ba9b348b Temporarily skip flakey test case
sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger

PR:		239292
Sponsored by:	The FreeBSD Foundation
2019-07-22 10:37:56 +00:00
Alan Somers
5a0b9a2776 fusefs: fix warnings in the tests reported by GCC
Sponsored by:	The FreeBSD Foundation
2019-07-20 05:21:13 +00:00
Alan Somers
fca79580be sendfile: don't panic when VOP_GETPAGES_ASYNC returns an error
PR:		236466
Sponsored by:	The FreeBSD Foundation
2019-07-19 18:03:30 +00:00
Alan Somers
ed74f781c9 fusefs: add a intr/nointr mount option
FUSE file systems can optionally support interrupting outstanding
operations.  However, the file system does not identify to the kernel at
mount time whether it's capable of doing that.  Instead it signals its
noncapability by returning ENOSYS to the first FUSE_INTERRUPT operation it
receives.  That's a problem for reliable signal delivery, because the kernel
must choose which thread should get a signal before it knows whether the
FUSE server can handle interrupts.  The problem is even worse because the
FUSE protocol allows a file system to simply ignore all FUSE_INTERRUPT
operations.

Fix the signal delivery logic by making interruptibility an opt-in mount
option.  This will require a corresponding change to libfuse, but not to
most file systems that link to libfuse.

Bump __FreeBSD_version due to the new mount option.

Sponsored by:	The FreeBSD Foundation
2019-07-18 17:55:13 +00:00
Alan Somers
d26d63a4af fusefs: multiple interruptility improvements
1) Don't explicitly not mask SIGKILL.  kern_sigprocmask won't allow it to be
   masked, anyway.

2) Fix an infinite loop bug.  If a process received both a maskable signal
   lower than 9 (like SIGINT) and then received SIGKILL,
   fticket_wait_answer would spin.  msleep would immediately return EINTR,
   but cursig would return SIGINT, so the sleep would get retried.  Fix it
   by explicitly checking whether SIGKILL has been received.

3) Abandon the sig_isfatal optimization introduced by r346357.  That
   optimization would cause fticket_wait_answer to return immediately,
   without waiting for a response from the server, if the process were going
   to exit anyway.  However, it's vulnerable to a race:

   1) fatal signal is received while fticket_wait_answer is sleeping.
   2) fticket_wait_answer sends the FUSE_INTERRUPT operation.
   3) fticket_wait_answer determines that the signal was fatal and returns
      without waiting for a response.
   4) Another thread changes the signal to non-fatal.
   5) The first thread returns to userspace.  Instead of exiting, the
      process continues.
   6) The application receives EINTR, wrongly believes that the operation
      was successfully interrupted, and restarts it.  This could cause
      problems for non-idempotent operations like FUSE_RENAME.

Reported by:    kib (the race part)
Sponsored by:   The FreeBSD Foundation
2019-07-17 22:45:43 +00:00
John Baldwin
32451fb9fc Add ptrace op PT_GET_SC_RET.
This ptrace operation returns a structure containing the error and
return values from the current system call.  It is only valid when a
thread is stopped during a system call exit (PL_FLAG_SCX is set).

The sr_error member holds the error value from the system call.  Note
that this error value is the native FreeBSD error value that has _not_
been translated to an ABI-specific error value similar to the values
logged to ktrace.

If sr_error is zero, then the return values of the system call will be
set in sr_retval[0] and sr_retval[1].

Reviewed by:	kib
MFC after:	1 month
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D20901
2019-07-15 21:48:02 +00:00
John Baldwin
c8ea87310c Add a test for PT_GET_SC_ARGS.
Reviewed by:	kib
MFC after:	1 month
Sponsored by:	DARPA
Differential Revision:	https://reviews.freebsd.org/D20899
2019-07-15 21:26:55 +00:00
Alan Somers
97b0512b23 projects/fuse2: build fixes
* Fix the kernel build with gcc by removing a redundant extern declaration
* In the tests, fix a printf format specifier that assumed LP64

Sponsored by:	The FreeBSD Foundation
2019-07-13 14:42:09 +00:00
Li-Wen Hsu
1db8307b66 Correct definitions in sys.opencrypto.runtests.main for 32bit platform
Reviewed by:	cem, jhb
MFC after:	3 days
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D20894
2019-07-10 01:08:08 +00:00
Li-Wen Hsu
43527153dc Skip sys.netpfil.pf.names.names and sys.netpfil.pf.synproxy.synproxy
temporarily because kernel panics when flushing epair queue.

PR:		238870
Sponsored by:	The FreeBSD Foundation
2019-06-29 12:19:57 +00:00
Alan Somers
7e1f5432f4 fusefs: don't leak memory of unsent operations on unmount
Sponsored by:	The FreeBSD Foundation
2019-06-28 18:48:02 +00:00
Alan Somers
7f49ce7a0b MFHead @349476
Sponsored by:	The FreeBSD Foundation
2019-06-27 23:50:54 +00:00
Alan Somers
435ecf40bb fusefs: recycle vnodes after their last unlink
Previously fusefs would never recycle vnodes.  After VOP_INACTIVE, they'd
linger around until unmount or the vnlru reclaimed them.  This commit
essentially actives and inlines the old reclaim_revoked sysctl, and fixes
some issues dealing with the attribute cache and multiply linked files.

Sponsored by:	The FreeBSD Foundation
2019-06-27 20:18:12 +00:00
Alan Somers
9cf5812603 fusefs: fix a memory leak in the forget test
Sponsored by:	The FreeBSD Foundation
2019-06-27 17:44:21 +00:00
Alan Somers
f74b33d9db fusefs: tighten expectations in mmap tests
In r349378 I fixed mmap's habit of reading more data than was available.

Sponsored by:	The FreeBSD Foundation
2019-06-26 23:10:20 +00:00
Alan Somers
7fc0921d7e fusefs: annotate deliberate file descriptor leaks in the tests
closing a file descriptor causes FUSE activity that is superfluous to the
purpose of most tests, but would nonetheless require matching expectations.
Rather than do that, most tests deliberately leak file descriptors instead.
This commit moves the leakage from each test into two trivial functions:
leak and leakdir.  Hopefully Coverity will only complain about those
functions and not all of their callers.

Sponsored by:	The FreeBSD Foundation
2019-06-26 20:25:57 +00:00
Alan Somers
c51f519b36 fusefs: run the io tests with direct io, too
Now the io tests are run in all cache modes.  The fusefs test suite can now
get adequate coverage without changing the value of
vfs.fusefs.data_cache_mode, which is only needed for legacy file systems
now.

Sponsored by:	The FreeBSD Foundation
2019-06-26 19:10:39 +00:00
Alan Somers
f8ebf1cd7e fusefs: implement protocol 7.23's FUSE_WRITEBACK_CACHE option
As of protocol 7.23, fuse file systems can specify their cache behavior on a
per-mountpoint basis.  If they set FUSE_WRITEBACK_CACHE in
fuse_init_out.flags, then they'll get the writeback cache.  If not, then
they'll get the writethrough cache.  If they set FOPEN_DIRECT_IO in every
FUSE_OPEN response, then they'll get no cache at all.

The old vfs.fusefs.data_cache_mode sysctl is ignored for servers that use
protocol 7.23 or later.  However, it's retained for older servers,
especially for those running in jails that lack access to the new protocol.

This commit also fixes two other minor test bugs:
* WriteCluster:SetUp was using an uninitialized variable.
* Read.direct_io_pread wasn't verifying that the cache was actually
  bypassed.

Sponsored by:	The FreeBSD Foundation
2019-06-26 17:32:31 +00:00
Alan Somers
fef464546c fusefs: implement the "time_gran" feature.
If a server supports a timestamp granularity other than 1ns, it can tell the
client this as of protocol 7.23.  The client will use that granularity when
updating its cached timestamps during write.  This way the timestamps won't
appear to change following flush.

Sponsored by:	The FreeBSD Foundation
2019-06-26 02:09:22 +00:00
Alan Somers
6efc53b9f3 fusefs: delete obsolete comments in the tests
I originally thought that the kernel would be responsible for ctime in
protocol 7.23.  But now I realize that's not the case.  The server is
responsible for ctime.  The kernel only sets it when there are dirty writes
cached, because that's when the server can't.

Sponsored by:	The FreeBSD Foundation
2019-06-26 00:06:41 +00:00
Alan Somers
0a8fe2d369 fusefs: set ctime during FUSE_SETATTR following a write
As of r349396 the kernel will internally update the mtime and ctime of files
on write.  It will also flush the mtime should a SETATTR happen before the
data cache gets flushed.  Now it will flush the ctime too, if the server is
using protocol 7.23 or higher.

This is the only case in which the kernel will explicitly set a file's
ctime, since neither utimensat(2) nor any other user interfaces allow it.

Sponsored by:	The FreeBSD Foundation
2019-06-26 00:03:37 +00:00
Alan Somers
788af9538a fusefs: automatically update mtime and ctime on write
Writing should implicitly update a file's mtime and ctime.  For fuse, the
server is supposed to do that.  But the client needs to do it too, because
the FUSE_WRITE response does not include time attributes, and it's not
desirable to issue a GETATTR after every WRITE.  When using the writeback
cache, there's another hitch: the kernel should ignore the mtime and ctime
fields in any GETATTR response for files with a dirty write cache.

Sponsored by:	The FreeBSD Foundation
2019-06-25 23:40:18 +00:00
Alan Somers
f2704f05cb fusefs: fix the tests for non-default values of MAXPHYS
Sponsored by:	The FreeBSD Foundation
2019-06-25 21:21:34 +00:00
Alan Somers
6ca3b02b1b fusefs: fix the tests for nondefault values of vfs.maxbcachebuf
Sponsored by:	The FreeBSD Foundation
2019-06-25 18:58:51 +00:00
Alan Somers
0d3a88d76c fusefs: writes should update the file size, even when data_cache_mode=0
Writes that extend a file should update the file's size.  r344185 restricted
that behavior for fusefs to only happen when the data cache was enabled.
That probably made sense at the time because the attribute cache wasn't
fully baked yet.  Now that it is, we should always update the cached file
size during write.

Sponsored by:	The FreeBSD Foundation
2019-06-25 18:36:11 +00:00
Alan Somers
b9e2019755 fusefs: rewrite vop_getpages and vop_putpages
Use the standard facilities for getpages and putpages instead of bespoke
implementations that don't work well with the writeback cache.  This has
several corollaries:

* Change the way we handle short reads _again_.  vfs_bio_getpages doesn't
  provide any way to handle unexpected short reads.  Plus, I found some more
  lock-order problems.  So now when the short read is detected we'll just
  clear the vnode's attribute cache, forcing the file size to be requeried
  the next time it's needed.  VOP_GETPAGES doesn't have any way to indicate
  a short read to the "caller", so we just bzero the rest of the page
  whenever a short read happens.

* Change the way we decide when to set the FUSE_WRITE_CACHE bit.  We now set
  it for clustered writes even when the writeback cache is not in use.

Sponsored by:   The FreeBSD Foundation
2019-06-25 17:24:43 +00:00
Alan Somers
48417ae0ba fusefs: fix multiple issues with the io tests
* During TearDown, close the test file before the backing file.  That way
  the backing file artifact will have the correct contents after the test
  completes.  It doesn't matter when running in Kyua, but it may when
  running the test manually.
* Add a closeopen operation that mimics what FSX does with the "-c" option.
* Skip mmap-related tests when vfs.fusefs.data_cache_mode == 0

Sponsored by:	The FreeBSD Foundation
2019-06-25 16:49:20 +00:00
Alan Somers
17575bad85 fusefs: improve the short read fix from r349279
VOP_GETPAGES intentionally tries to read beyond EOF, so fuse_read_biobackend
can't rely on bp->b_resid > 0 indicating a short read.  And adjusting
bp->b_count after a short read seems to cause some sort of resource leak.
Instead, store the shortfall in the bp->b_fsprivate1 field.

Sponsored by:	The FreeBSD Foundation
2019-06-24 17:05:31 +00:00
Li-Wen Hsu
01e92e2977 Skip sys.netinet.socket_afinet.socket_afinet_bind_zero temporarily because it
doesn't work when mac_portacl(4) loaded

PR:		238781
Sponsored by:	The FreeBSD Foundation
2019-06-23 19:37:12 +00:00
Alan Somers
aef22f2d75 fusefs: correctly handle short reads
A fuse server may return a short read for three reasons:

* The file is opened with FOPEN_DIRECT_IO.  In this case, the short read
  should be returned directly to userland.  We already handled this case
  correctly.

* The file was truncated server-side, and the read hit EOF.  In this case,
  the kernel should update the file size.  Fixed in the case of VOP_READ.
  Fixing this for VOP_GETPAGES is TODO.

* The file is opened in writeback mode, there are dirty buffers past what
  the server thinks is the file's EOF, and the read hit what the server
  thinks is the file's EOF.  In this case, the client is trying to read a
  hole, and should zero-fill it.  We already handled this case, and I added
  a test for it.

Sponsored by:	The FreeBSD Foundation
2019-06-21 21:44:31 +00:00
Alan Somers
87ff949a7b fusefs: raise protocol level to 7.23
None of the new features are implemented yet.  This commit just adds the new
protocol definitions and adds backwards-compatibility code for pre 7.23
servers.

Sponsored by:	The FreeBSD Foundation
2019-06-21 04:57:23 +00:00
Alan Somers
1f309e37f5 fusefs: update tests after r349260
r349260 removed some Linuxisms from the FUSE protocol header file in favor
of standard C99 types.  This change follows suit in the tests.

Sponsored by:	The FreeBSD Foundation
2019-06-21 04:37:11 +00:00
Alan Somers
7cbb8e8a06 fusefs: raise protocol level to 7.15
This protocol level adds two new features: the ability for the server to
store or retrieve data into/from the client's cache.  But the messages
aren't defined soundly since they identify the file only by its inode,
without the generation number.  So it's possible for them to modify the
wrong file's cache.  Also, I don't know of any file systems in ports that
use these messages.  So I'm not implementing them.  I did add a (disabled)
test for the store message, however.

Sponsored by:	The FreeBSD Foundation
2019-06-20 23:32:25 +00:00
Alan Somers
a1c9f4ad0d fusefs: implement VOP_BMAP
If the fuse daemon supports FUSE_BMAP, then use that for the block mapping.
Otherwise, use the same technique used by vop_stdbmap.  Report large values
for runp and runb in order to maximize read clustering and minimize upcalls,
even if we don't know the true layout.

The major result of this change is that sequential reads to FUSE files will
now usually happen 128KB at a time instead of 64KB.

Sponsored by:	The FreeBSD Foundation
2019-06-20 17:08:21 +00:00
Alan Somers
e532a99901 MFHead @349234
Sponsored by:	The FreeBSD Foundation
2019-06-20 15:56:08 +00:00
Alan Somers
84879e46c2 fusefs: multiple fixes related to the write cache
* Don't always write the last page synchronously.  That's not actually
  required.  It was probably just masking another bug that I fixed later,
  possibly in r349021.

* Enable the NotifyWriteback tests now that Writeback cache is working.

* Add a test to ensure that the write cache isn't flushed synchronously when
  in writeback mode.

Sponsored by:	The FreeBSD Foundation
2019-06-17 23:34:11 +00:00
Alan Somers
0482ec3e3f fusefs: run the Io tests with various combinations of mount options
Sponsored by:	The FreeBSD Foundation
2019-06-17 22:13:59 +00:00
Alan Somers
402b609c80 fusefs: use cluster_read for more readahead
fusefs will now use cluster_read.  This allows readahead of more than one
cache block.  However, it won't yet actually cluster the reads because that
requires VOP_BMAP, which fusefs does not yet implement.

Sponsored by:	The FreeBSD Foundation
2019-06-17 22:01:23 +00:00
Conrad Meyer
179f62805c random(4): Fortuna: allow increased concurrency
Add experimental feature to increase concurrency in Fortuna.  As this
diverges slightly from canonical Fortuna, and due to the security
sensitivity of random(4), it is off by default.  To enable it, set the
tunable kern.random.fortuna.concurrent_read="1".  The rest of this commit
message describes the behavior when enabled.

Readers continue to update shared Fortuna state under global mutex, as they
do in the status quo implementation of the algorithm, but shift the actual
PRF generation out from under the global lock.  This massively reduces the
CPU time readers spend holding the global lock, allowing for increased
concurrency on SMP systems and less bullying of the harvestq kthread.

It is somewhat of a deviation from FS&K.  I think the primary difference is
that the specific sequence of AES keys will differ if READ_RANDOM_UIO is
accessed concurrently (as the 2nd thread to take the mutex will no longer
receive a key derived from rekeying the first thread).  However, I believe
the goals of rekeying AES are maintained: trivially, we continue to rekey
every 1MB for the statistical property; and each consumer gets a
forward-secret, independent AES key for their PRF.

Since Chacha doesn't need to rekey for sequences of any length, this change
makes no difference to the sequence of Chacha keys and PRF generated when
Chacha is used in place of AES.

On a GENERIC 4-thread VM (so, INVARIANTS/WITNESS, numbers not necessarily
representative), 3x concurrent AES performance jumped from ~55 MiB/s per
thread to ~197 MB/s per thread.  Concurrent Chacha20 at 3 threads went from
roughly ~113 MB/s per thread to ~430 MB/s per thread.

Prior to this change, the system was extremely unresponsive with 3-4
concurrent random readers; each thread had high variance in latency and
throughput, depending on who got lucky and won the lock.  "rand_harvestq"
thread CPU use was high (double digits), seemingly due to spinning on the
global lock.

After the change, concurrent random readers and the system in general are
much more responsive, and rand_harvestq CPU use dropped to basically zero.

Tests are added to the devrandom suite to ensure the uint128_add64 primitive
utilized by unlocked read functions to specification.

Reviewed by:	markm
Approved by:	secteam(delphij)
Relnotes:	yes
Differential Revision:	https://reviews.freebsd.org/D20313
2019-06-17 20:29:13 +00:00
Xin LI
f89d207279 Separate kernel crc32() implementation to its own header (gsb_crc32.h) and
rename the source to gsb_crc32.c.

This is a prerequisite of unifying kernel zlib instances.

PR:		229763
Submitted by:	Yoshihiro Ota <ota at j.email.ne.jp>
Differential Revision:	https://reviews.freebsd.org/D20193
2019-06-17 19:49:08 +00:00
Alan Somers
6fa772a88e fusefs: skip the Write.mmap test when mmap is not available
fusefs doesn't not allow mmap when data caching is disabled.

Sponsored by:	The FreeBSD Foundation
2019-06-17 17:17:01 +00:00
Alan Somers
d569012f45 fusefs: implement non-clustered readahead
fusefs will now read ahead at most one cache block at a time (usually 64
KB).  Clustered reads are still TODO.  Individual file systems may disable
read ahead by setting fuse_init_out.max_readahead=0 during initialization.

Sponsored by:	The FreeBSD Foundation
2019-06-17 16:56:51 +00:00
Conrad Meyer
d0d71d818c random(4): Generalize algorithm-independent APIs
At a basic level, remove assumptions about the underlying algorithm (such as
output block size and reseeding requirements) from the algorithm-independent
logic in randomdev.c.  Chacha20 does not have many of the restrictions that
AES-ICM does as a PRF (Pseudo-Random Function), because it has a cipher
block size of 512 bits.  The motivation is that by generalizing the API,
Chacha is not penalized by the limitations of AES.

In READ_RANDOM_UIO, first attempt to NOWAIT allocate a large enough buffer
for the entire user request, or the maximal input we'll accept between
signal checking, whichever is smaller.  The idea is that the implementation
of any randomdev algorithm is then free to divide up large requests in
whatever fashion it sees fit.

As part of this, two responsibilities from the "algorithm-generic" randomdev
code are pushed down into the Fortuna ra_read implementation (and any other
future or out-of-tree ra_read implementations):

  1. If an algorithm needs to rekey every N bytes, it is responsible for
  handling that in ra_read(). (I.e., Fortuna's 1MB rekey interval for AES
  block generation.)

  2. If an algorithm uses a block cipher that doesn't tolerate partial-block
  requests (again, e.g., AES), it is also responsible for handling that in
  ra_read().

Several APIs are changed from u_int buffer length to the more canonical
size_t.  Several APIs are changed from taking a blockcount to a bytecount,
to permit PRFs like Chacha20 to directly generate quantities of output that
are not multiples of RANDOM_BLOCKSIZE (AES block size).

The Fortuna algorithm is changed to NOT rekey every 1MiB when in Chacha20
mode (kern.random.use_chacha20_cipher="1").  This is explicitly supported by
the math in FS&K §9.4 (Ferguson, Schneier, and Kohno; "Cryptography
Engineering"), as well as by their conclusion: "If we had a block cipher
with a 256-bit [or greater] block size, then the collisions would not
have been an issue at all."

For now, continue to break up reads into PAGE_SIZE chunks, as they were
before.  So, no functional change, mostly.

Reviewed by:	markm
Approved by:	secteam(delphij)
Differential Revision:	https://reviews.freebsd.org/D20312
2019-06-17 15:09:12 +00:00
Conrad Meyer
403c041316 random(4): Add regression tests for uint128 implementation, Chacha CTR
Add some basic regression tests to verify behavior of both uint128
implementations at typical boundary conditions, to run on all architectures.

Test uint128 increment behavior of Chacha in keystream mode, as used by
'kern.random.use_chacha20_cipher=1' (r344913) to verify assumptions at edge
cases.  These assumptions are critical to the safety of using Chacha as a
PRF in Fortuna (as implemented).

(Chacha's use in arc4random is safe regardless of these tests, as it is
limited to far less than 4 billion blocks of output in that API.)

Reviewed by:	markm
Approved by:	secteam(gordon)
Differential Revision:	https://reviews.freebsd.org/D20392
2019-06-17 14:59:45 +00:00
Alan Somers
eadd12d35d fusefs: rename the ReadCacheable.default_readahead test
The test didn't actually have anything to do with readahead.  Rename it to
"ReadCacheable.cache_block"

Sponsored by:	The FreeBSD Foundation
2019-06-17 14:42:27 +00:00
Alan Somers
b5aaf286ea fusefs: fix the "write-through" of write-through cacheing
Our fusefs(5) module supports three cache modes: uncached, write-through,
and write-back.  However, the write-through mode (which is the default) has
never actually worked as its name suggests.  Rather, it's always been more
like "write-around".  It wrote directly, bypassing the cache.  The cache
would only be populated by a subsequent read of the same data.

This commit fixes that problem.  Now the write-through mode works as one
would expect: write(2) immediately adds data to the cache and then blocks
while the daemon processes the write operation.

A side effect of this change is that non-cache-block-aligned writes will now
incur a read-modify-write cycle of the cache block.  The old behavior
(bypassing write cache entirely) can still be achieved by opening a file
with O_DIRECT.

PR:		237588
Sponsored by:	The FreeBSD Foundation
2019-06-14 19:47:48 +00:00
Alan Somers
8eecd9ce05 fusefs: enable write clustering
Enable write clustering in fusefs whenever cache mode is set to writeback
and the "async" mount option is used.  With default values for MAXPHYS,
DFLTPHYS, and the fuse max_write mount parameter, that means sequential
writes will now be written 128KB at a time instead of 64KB.

Also, add a regression test for PR 238565, a panic during unmount that
probably affects UFS, ext2, and msdosfs as well as fusefs.

PR:		238565
Sponsored by:	The FreeBSD Foundation
2019-06-14 18:14:51 +00:00
Alan Somers
dff3a6b410 fusefs: fix a bug with WriteBack cacheing
An errant vfs_bio_clrbuf snuck in in r348931.  Surprisingly, it doesn't have
any effect most of the time.  But under some circumstances it cause the
buffer to behave in a write-only fashion.

Sponsored by:	The FreeBSD Foundation
2019-06-13 19:07:03 +00:00
Alan Somers
38edd595df Add test cases for epair
Implements the missing test cases for epair in a similar fashion to the
existing tests. Fixes shared abstractions to work with epair tests.

Submitted by:	Ryan Moeller <ryan@freqlabs.com>
Reviewed by:	asomers
MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D20498
2019-06-13 05:05:58 +00:00
Alan Somers
93c0c1d4ce fusefs: fix a page fault with writeback cacheing
When truncating a file downward through a dirty buffer, it's neccessary to
update the buffer's b->dirtyend.

Sponsored by:	The FreeBSD Foundation
2019-06-11 23:46:31 +00:00
Alan Somers
a87e0831ab fusefs: WIP fixing writeback cacheing
The current "writeback" cache mode, selected by the
vfs.fusefs.data_cache_mode sysctl, doesn't do writeback cacheing at all.  It
merely goes through the motions of using buf(9), but then writes every
buffer synchronously.  This commit:

* Enables delayed writes when the sysctl is set to writeback cacheing
* Fixes a cache-coherency problem when extending a file whose last page has
  just been written.
* Removes the "sync" mount option, which had been set unconditionally.
* Adjusts some SDT probes
* Adds several new tests that mimic what fsx does but with more control and
  without a real file system.  As I discover failures with fsx, I add
  regression tests to this file.
* Adds a test that ensures we can append to a file without reading any data
  from it.

This change is still incomplete.  Clustered writing is not yet supported,
and there are frequent "panic: vm_fault_hold: fault on nofault entry" panics
that I need to fix.

Sponsored by:	The FreeBSD Foundation
2019-06-11 16:32:33 +00:00
Alan Somers
b690d120d9 fusefs: fix an infinite loop in the tests
It was possible for the MockFS thread to infinitely loop if it got an error
reading from /dev/fuse.

Sponsored by:	The FreeBSD Foundation
2019-06-11 16:16:16 +00:00
Alan Somers
d53a96f16f fusefs: fix a comment. No functional change.
Sponsored by:	The FreeBSD Foundation
2019-06-10 22:23:37 +00:00
Alan Somers
2d6bf515df fusefs: add some explicit tests for FUSE_FORGET
Sponsored by:	The FreeBSD Foundation
2019-06-06 16:29:08 +00:00
Alan Somers
0269ae4c19 MFHead @348740
Sponsored by:	The FreeBSD Foundation
2019-06-06 16:20:50 +00:00
Alan Somers
d2654efffe fusefs: remove debugging code that accidentally snuck into r348365
Sponsored by:	The FreeBSD Foundation
2019-06-04 19:06:24 +00:00
Alan Somers
a639731ba9 fusefs: respect RLIMIT_FSIZE
Sponsored by:	The FreeBSD Foundation
2019-06-03 23:24:07 +00:00
Alan Somers
6ff7f297f8 fusefs: don't require FUSE_EXPORT_SUPPORT for async invalidation
In r348560 I thought that FUSE_EXPORT_SUPPORT was required for cases where
the node to be invalidated (or the parent of the entry to be invalidated)
wasn't cached.  But I realize now that that's not the case.  During entry
invalidation, if the parent isn't in the vfs hash table, then it must've
been reclaimed.  And since fuse_vnop_reclaim does a cache_purge, that means
the entry to be invalidated has already been removed from the namecache.
And during inode invalidation, if the inode to be invalidated isn't in the
vfs hash table, then it too must've been reclaimed.  In that case it will
have no buffer cache to invalidate.

Sponsored by:	The FreeBSD Foundation
2019-06-03 20:45:32 +00:00
Alan Somers
eae1ae132c fusefs: support asynchronous cache invalidation
Protocol 7.12 adds a way for the server to notify the client that it should
invalidate an inode's data cache and/or attributes.  This commit implements
that mechanism.  Unlike Linux's implementation, ours requires that the file
system also supports FUSE_EXPORT_SUPPORT (NFS-style lookups).  Otherwise the
invalidation operation will return EINVAL.

Sponsored by:	The FreeBSD Foundation
2019-06-03 17:34:01 +00:00
Alan Somers
c2d70d6e6f fusefs: support name cache invalidation
Protocol 7.12 adds a way for the server to notify the client that it should
invalidate an entry from its name cache.  This commit implements that
mechanism.

Sponsored by:	The FreeBSD Foundation
2019-06-01 00:11:19 +00:00
Alan Somers
0d2bf48996 fusefs: check the vnode cache when looking up files for the NFS server
FUSE allows entries to be cached for a limited amount of time.  fusefs's
vnop_lookup method already implements that using the timeout functionality
of cache_lookup/cache_enter_time.  However, lookups for the NFS server go
through a separate path: vfs_vget.  That path can't use the same timeout
functionality because cache_lookup/cache_enter_time only work on pathnames,
whereas vfs_vget works by inode number.

This commit adds entry timeout information to the fuse vnode structure, and
checks it during vfs_vget.  This allows the NFS server to take advantage of
cached entries.  It's also the same path that FUSE's asynchronous cache
invalidation operations will use.

Sponsored by:	The FreeBSD Foundation
2019-05-31 21:22:58 +00:00
Alan Somers
a34cdd26d0 fusefs: prefer FUSE_ROOT_ID to literal 1 in the tests
Sponsored by:	The FreeBSD Foundation
2019-05-31 17:02:37 +00:00
Li-Wen Hsu
dff3f7f89b Remove tests for the deprecated algorithms in r348206
The tests are failing because the return value and output have changed, but
before test code structure adjusted, removing these test cases help people
be able to focus on more important cases.

Discussed with:	emaste
MFC with:	r348206
Sponsored by:	The FreeBSD Foundation
2019-05-31 04:29:29 +00:00
Alan Somers
a4856c96d0 fusefs: raise protocol level to 7.12
This commit raises the protocol level and adds backwards-compatibility code
to handle structure size changes.  It doesn't implement any new features.
The new features added in protocol 7.12 are:

* server-side umask processing (which FreeBSD won't do)
* asynchronous inode and directory entry invalidation (which I'll do next)

Sponsored by:	The FreeBSD Foundation
2019-05-29 16:39:52 +00:00
Alan Somers
d4fd0c8148 fusefs: set the flags fields of fuse_write_in and fuse_read_in
These fields are supposed to contain the file descriptor flags as supplied
to open(2) or set by fcntl(2).  The feature is kindof useless on FreeBSD
since we don't supply all of these flags to fuse (because of the weak
relationship between struct file and struct vnode).  But we should at least
set the access mode flags (O_RDONLY, etc).

This is the last fusefs change needed to get full protocol 7.9 support.
There are still a few options we don't support for good reason (mandatory
file locking is dumb, flock support is broken in the protocol until 7.17,
etc), but there's nothing else to do at this protocol level.

Sponsored by:	The FreeBSD Foundation
2019-05-28 01:09:19 +00:00
Alan Somers
8aa24ed381 fusefs: flock(2) locks must be implemented in-kernel
If a FUSE file system sets the FUSE_POSIX_LOCKS flag then it can support
fcntl(2)-style locks directly.  However, the protocol does not adequately
support flock(2)-style locks until revision 7.17.  They must be implemented
locally in-kernel instead.  This unfortunately breaks the interoperability
of fcntl(2) and flock(2) locks for file systems that support the former.
C'est la vie.

Prior to this commit flock(2) would get sent to the server as a
fcntl(2)-style lock with the lock owner field set to stack garbage.

Sponsored by:	The FreeBSD Foundation
2019-05-28 00:03:46 +00:00
Alan Somers
b7774b82cb fusefs: fix an alignment issue in the tests on arm
Sponsored by:   The FreeBSD Foundation
2019-05-27 21:51:43 +00:00
Alan Somers
bda39894c5 fusefs: set FUSE_WRITE_CACHE when writing from cache
This bit tells the server that we're not sure which uid, gid, and/or pid
originated the write.  I don't know of a single file system that cares, but
it's part of the protocol.

Sponsored by:	The FreeBSD Foundation
2019-05-27 21:36:28 +00:00
Alan Somers
7637cc62ab fusefs: remove obsolete comments in the tests
Sponsored by:	The FreeBSD Foundation
2019-05-27 17:14:46 +00:00
Alan Somers
29edc611c1 fusefs: make the tests more cplusplusy
* Prefer std::unique_ptr to raw pointers
* Prefer pass-by-reference to pass-by-pointer
* Prefer static_cast to C-style cast, unless it's too much typing

Reported by:	ngie
Sponsored by:	The FreeBSD Foundation
2019-05-27 17:08:16 +00:00
Alan Somers
cc04566c46 fusefs: more build fixes
* Fix printf format strings on 32-bit OSes
* Fix -Wclass-memaccess violation on GCC-8 caused by using memset on an object
  of non-trivial type.
* Fix memory leak in MockFS::init
* Fix -Wcast-align error on i386 in expect_readdir
* Fix some heterogenous comparison errors on 32-bit OSes.

Sponsored by:	The FreeBSD Foundation
2019-05-26 03:52:35 +00:00
Alan Somers
93fecd02a1 fusefs: misc build fixes
* Only build the tests on platforms with C++14 support
* Fix an undefined symbol error on lint builds
* Remove an unused function: fiov_clear

Sponsored by:	The FreeBSD Foundation
2019-05-25 21:40:27 +00:00
Alan Somers
e97ae4ad2d fusefs: implement FUSE_ASYNC_READ
If a daemon sets the FUSE_ASYNC_READ flag during initialization, then the
client is allowed to issue multiple concurrent reads for the same file
handle.  Otherwise concurrent reads are not allowed.  This commit implements
it.  Previously we unconditionally disallowed concurrent reads.

Sponsored by:	The FreeBSD Foundation
2019-05-24 05:12:43 +00:00
Alan Somers
e5b50fe736 fusefs: Make fuse file systems NFS-exportable
This commit adds the VOPs needed by userspace NFS servers (tested with
net/unfs3).  More work is needed to make the in-kernel nfsd work, because of
its stateless nature.  It doesn't open files prior to doing I/O.  Also, the
NFS-related VOPs currently ignore the entry cache.

Sponsored by:	The FreeBSD Foundation
2019-05-23 00:44:01 +00:00
Alan Somers
18a2264e27 fusefs: fix "recursing on non recursive lockmgr" panic
When mounted with -o default_permissions and when
vfs.fusefs.data_cache_mode=2, fuse_io_strategy would try to clear the suid
bit after a successful write by a non-owner.  When combined with a
not-yet-committed attribute-caching patch I'm working on, and if the
FUSE_SETATTR response indicates an unexpected filesize (legal, if the file
system has other clients), this would end up calling vtruncbuf.  That would
panic, because the buffer lock was already held by bufwrite or bufstrategy
or something else upstack from fuse_vnop_strategy.

Sponsored by:	The FreeBSD Foundation
2019-05-22 23:30:51 +00:00
Bjoern A. Zeeb
aa32159686 Add very basic afinet socket tests which I started to write in order
to then try to reproduce a kernel panic, which turned out to be a
race condition and hard to test from here.

Commit the changes anywhere as the "bind zero" case was a surprise
to me and we should try to maintain this status.

Also it is easy examples someone can build upon.

With help from:	markj
Event:		Waterloo Hackathon 2019
2019-05-21 19:42:04 +00:00
Alan Somers
a6fac00c53 fusefs: Allow update mounts
Allow "mount -u" to change some mount options for fusefs.

Sponsored by:	The FreeBSD Foundation
2019-05-21 19:34:39 +00:00
Enji Cooper
f2a344455f Add my name to the copyright
I have contributed a number of changes to these tests over the past few
hundred revisions, and believe I deserve credit for the changes I have
made (plus, the copyright hadn't been updated since 2014).

MFC after:	1 week
2019-05-21 04:11:16 +00:00
Enji Cooper
ef02523dc8 Follow up to r348042: cast aad to a byte array
This is not completely necessary today, but this change is being made in a
conservative manner to avoid accidental breakage in the future, if this ever
was a unicode string.

PR:		237403
MFC after:	1 week
2019-05-21 04:03:22 +00:00
Enji Cooper
ffbc8cc0f2 Fix encoding issues with python 3
In python 3, the default encoding was switched from ascii character sets to
unicode character sets in order to support internationalization by default.
Some interfaces, like ioctls and packets, however, specify data in terms of
non-unicode encodings formats, either in host endian (`fcntl.ioctl`) or
network endian (`dpkt`) byte order/format.

This change alters assumptions made by previous code where it was all
data objects were assumed to be basestrings, when they should have been
treated as byte arrays. In order to achieve this the following are done:
* str objects with encodings needing to be encoded as ascii byte arrays are
  done so via `.encode("ascii")`. In order for this to work on python 3 in a
  type agnostic way (as it anecdotally varied depending on the caller), call
  `.encode("ascii")` only on str objects with python 3 to cast them to ascii
  byte arrays in a helper function name `str_to_ascii(..)`.
* `dpkt.Packet` objects needing to be passed in to `fcntl.ioctl(..)` are done
  so by casting them to byte arrays via `bytes()`, which calls
  `dpkt.Packet__str__` under the covers and does the necessary str to byte array
  conversion needed for the `dpkt` APIs and `struct` module.

In order to accomodate this change, apply the necessary typecasting for the
byte array literal in order to search `fop.name` for nul bytes.

This resolves all remaining python 2.x and python 3.x compatibility issues on
amd64. More work needs to be done for the tests to function with i386, in
general (this is a legacy issue).

PR:		237403
MFC after:	1 week
Tested with:	python 2.7.16 (amd64), python 3.6.8 (amd64)
2019-05-21 03:52:48 +00:00
Enji Cooper
f6d7fcda27 Remove spurious newline
Even though some python styles suggest there should be multiple newlines between
methods/classes, for consistency with the surrounding code, it's best to be
consistent by having merely one newline between each functional block.

MFC after:	1 week
2019-05-21 02:49:15 +00:00
Enji Cooper
a60d9a9892 Fix KAT(CCM)?Parser file descriptor leaks
Make `KAT(CCM)?Parser` into a context suite-capable object by implementing
`__enter__` and `__exit__` methods which manage opening up the file descriptors
and closing them on context exit. This implementation was decided over adding
destructor logic to a `__del__` method, as there are a number of issues around
object lifetimes when dealing with threading cleanup, atexit handlers, and a
number of other less obvious edgecases. Plus, the architected solution is more
pythonic and clean.

Complete the iterator implementation by implementing a `__next__` method for
both classes which handles iterating over the data using a generator pattern,
and by changing `__iter__` to return the object instead of the data which it
would iterate over. Alias the `__next__` method to `next` when working with
python 2.x in order to maintain functional compatibility between the two major
versions.

As part of this work and to ensure readability, push the initialization of the
parser objects up one layer and pass it down to a helper function. This could
have been done via a decorator, but I was trying to keep it simple for other
developers to make it easier to modify in the future.

This fixes ResourceWarnings with python 3.

PR:		237403
MFC after:	1 week
Tested with:	python 2.7.16 (amd64), python 3.6.8 (amd64)
2019-05-21 02:30:43 +00:00
Enji Cooper
8c02634818 Squash deprecation warning related to array.array(..).tostring()
In version 3.2+, `array.array(..).tostring()` was renamed to
`array.array(..).tobytes()`. Conditionally call `array.array(..).tobytes()` if
the python version is 3.2+.

PR:		237403
MFC after:	1 week
2019-05-21 02:13:46 +00:00
Enji Cooper
e8b4bbdfae Followup to r347996
Replace uses of `foo.encode("hex")` with `binascii.hexlify(foo)` for forwards
compatibility between python 2.x and python 3.

PR:		237403
MFC after:	1 week
2019-05-21 00:30:29 +00:00
Enji Cooper
1466140817 Allow the end-user to pass along arguments to cryptotest.py via $CRYPTOTEST_ARGS
This allows someone to use `-v` to dump out standard output.
2019-05-20 22:32:26 +00:00
Edward Tomasz Napierala
849f6862ff Improve tree(3) tests by using ATF_REQUIRE where applicable.
MFC after:	2 weeks
Sponsored by:	Klara Inc.
2019-05-20 18:35:23 +00:00
Enji Cooper
d99c2cecc8 Replace uses of foo.(de|en)code('hex') with binascii.(un)?hexlify(foo)
Python 3 no longer doesn't support encoding/decoding hexadecimal numbers using
the `str.format` method. The backwards compatible new method (using the
binascii module/methods) is a comparable means of converting to/from
hexadecimal format.

In short, the functional change is the following:
* `foo.decode('hex')` -> `binascii.unhexlify(foo)`
* `foo.encode('hex')` -> `binascii.hexlify(foo)`

While here, move the dpkt import in `cryptodev.py` down per PEP8, so it comes
after the standard library provided imports.

PR:		237403
MFC after:	1 week
2019-05-20 16:38:12 +00:00
Alan Somers
fe221e0177 fusefs: forward UTIME_NOW to the server
If a user sets both atime and mtime to UTIME_NOW when calling a syscall like
utimensat(2), allow the server to choose what "now" means.  Due to the
design of FreeBSD's VFS, it's not possible to do this for just one of atime
or mtime; it's all or none.

PR:		237181
Sponsored by:	The FreeBSD Foundation
2019-05-16 23:17:39 +00:00
Alan Somers
e7f73af118 fusefs: allow the server to specify st_blksize
If the server sets fuse_attr.blksize to a nonzero value in the response to
FUSE_GETATTR, then the client should use that as the value for
stat.st_blksize .

Sponsored by:	The FreeBSD Foundation
2019-05-16 22:50:04 +00:00
Alan Somers
16bd2d47c7 fusefs: Upgrade FUSE protocol to version 7.9.
This commit upgrades the FUSE API to protocol 7.9 and adds unit tests for
backwards compatibility with servers built for version 7.8.  It doesn't
implement any of 7.9's new features yet.

Sponsored by:	The FreeBSD Foundation
2019-05-16 17:24:11 +00:00
Kristof Provost
b685454a11 ipsec tests: Skip if ipsec.ko is not loaded
As of r347410 IPSec is no longer built into GENERIC. The ipsec.ko module must
be loaded before we can execute the IPSec tests.

Check this, and skip the tests if IPSec is not available.
2019-05-16 02:18:57 +00:00
Alan Somers
77fbe6944f fusefs: fix more intermittency in the dev_fuse_poll tests
When using poll, kevent, or select there was a race window during which it
would be impossible to shut down the daemon.  The problem was that poll,
kevent, and select don't return when the file descriptor gets closed (or
maybe it was that the file descriptor got closed before those syscalls were
entered?).  The solution is to impose a timeout on those syscalls, and check
m_quit after they time out.

Sponsored by:	The FreeBSD Foundation
2019-05-15 20:01:41 +00:00
Alan Somers
6fd2d8e148 fusefs: fix some intermittency in the Kqueue.data test
Expect the FUSE_GETATTR operations for bar and baz to come in either order.

Sponsored by:	The FreeBSD Foundation
2019-05-15 19:23:29 +00:00
Alan Somers
c4fbda2b2c fusefs: commit missing file from r347547
Sponsored by:	The FreeBSD Foundation
2019-05-13 19:48:57 +00:00
Alan Somers
7648bc9fee MFHead @347527
Sponsored by:	The FreeBSD Foundation
2019-05-13 18:25:55 +00:00
Alan Somers
0a7c63e075 fusefs: Report the number of available ops in kevent(2)
Just like /dev/devctl, /dev/fuse will now report the number of operations
available for immediate read in the kevent.data field during kevent(2).

Sponsored by:	The FreeBSD Foundation
2019-05-12 15:27:18 +00:00
Alan Somers
3429092cd1 fusefs: support kqueue for /dev/fuse
/dev/fuse was already pollable with poll and select.  Add support for
kqueue, too.  And add tests for polling with poll, select, and kqueue.

Sponsored by:	The FreeBSD Foundation
2019-05-11 22:58:25 +00:00
Alan Somers
a81776c270 fusefs: fix intermittency in the interrupt tests
* In the fatal_signal test, wait for the daemon to receive FUSE_INTERRUPT
  before exiting.
* Explicitly disable restarting syscalls after SIGUSR2.  This fixes
  intermittency in the priority test.  I don't know why, but sometimes that
  test's mkdir would be restarted, and sometimes it would return EINTR.
  ERESTART should be the default.
* Remove a useless copy/pasted sleep in the priority test.

Sponsored by:	The FreeBSD Foundation
2019-05-10 18:18:41 +00:00
Alan Somers
99cf7bff46 fusefs: debugability improvements in the tests
Fix a mislocated statement from r347431, and add more detail for FUSE_MKDIR

Sponsored by:	The FreeBSD Foundation
2019-05-10 18:14:39 +00:00
Alan Somers
81a619c4e1 fusefs: fix intermittency in the Destroy.ok test
The handler for FUSE_DESTROY must shut down the daemon.

Sponsored by:	The FreeBSD Foundation
2019-05-10 16:58:05 +00:00
Alan Somers
7e0aac2408 fusefs: return ENOTCONN instead of EIO if the daemon dies suddenly
If the daemon dies, return ENOTCONN for all operations that have already
been sent to the daemon, as well as any new ones.

Sponsored by:	The FreeBSD Foundation
2019-05-10 16:41:33 +00:00
Alan Somers
fd182076ff fusefs: fix intermittency in the Interrupt.already_complete test
Sponsored by:	The FreeBSD Foundation
2019-05-10 15:55:30 +00:00
Alan Somers
8b73a4c5ae fusefs: fix running multiple daemons concurrently
When a FUSE daemon dies or closes /dev/fuse, all of that daemon's pending
requests must be terminated.  Previously that was done in /dev/fuse's
.d_close method.  However, d_close only gets called on the *last* close of
the device.  That means that if multiple daemons were running concurrently,
all but the last daemon to close would leave their I/O hanging around.  The
problem was easily visible just by running "kyua -v parallelism=2 test" in
fusefs's test directory.

Fix this bug by terminating a daemon's pending I/O during /dev/fuse's
cdvpriv dtor method instead.  That method runs on every close of a file.

Also, fix some potential races in the tests:
* Clear SA_RESTART when registering the daemon's signal handler so read(2)
  will return EINTR.
* Wait for the daemon to die before unmounting the mountpoint, so we won't
  see an unwanted FUSE_DESTROY operation in the mock file system.

Sponsored by:	The FreeBSD Foundation
2019-05-10 15:02:29 +00:00
Edward Tomasz Napierala
16e55b9e0e Try to unbreak the build after r347425.
MFC after:	2 weeks
2019-05-10 08:16:29 +00:00
Edward Tomasz Napierala
36a040183e Add simple regression tests for tree(3). Those are ATF-ified versions
of OpenBSD's regress/sys/sys/tree/.

Reviewed by:	ngie
MFC after:	2 weeks
Sponsored by:	Klara Inc.
Differential Revision:	https://reviews.freebsd.org/D20186
2019-05-10 07:46:14 +00:00
Enji Cooper
16f35864df Refactor tests/sys/opencrypto/runtests
* Convert from plain to TAP for slightly improved introspection when skipping
  the tests due to requirements not being met.
* Test for the net/py-dpkt (origin) package being required when running the
  tests, instead of relying on a copy of the dpkt.py module from 2014. This
  enables the tests to work with py3. Subsequently, remove
  `tests/sys/opencrypto/dpkt.py(c)?` via `make delete-old`.
* Parameterize out `python2` as `$PYTHON`.

PR:		237403
MFC after:	1 week
2019-05-10 00:03:32 +00:00
Alan Somers
a87257ac25 fusefs: shorten and consolidate sleeps
Some fusefs tests must sleep because they deliberately trigger a race, or
because they're testing the cache timeout functionality.  Consolidate the
sleep interval in a single place so it will be easy to adjust.  Shorten it
from either 500ms or 250ms to 100ms.  From experiment I find that 10ms works
every time, so 100ms should be fairly safe.

Sponsored by:	The FreeBSD Foundation
2019-05-09 18:23:09 +00:00
Alan Somers
f528b38f60 fusefs: eliminate some sleeps in the Interrupt tests
Replace some sleeps with semaphore operations.  Not all sleeps can be
replaced, though.  Some are trying to lose a race.

Sponsored by:	The FreeBSD Foundation
2019-05-09 17:57:04 +00:00
Alan Somers
d5ff268834 fusefs: create sockets with FUSE_MKNOD, not FUSE_CREATE
libfuse expects sockets to be created with FUSE_MKNOD, not FUSE_CREATE,
because that's how Linux does it.  My first attempt at creating sockets
(r346894) used FUSE_CREATE because FreeBSD uses VOP_CREATE for this purpose.
There are no backwards-compatibility concerns with this change, because
socket support hasn't yet been merged to head.

Sponsored by:	The FreeBSD Foundation
2019-05-09 16:25:01 +00:00
Alan Somers
002e54b0aa fusefs: clear a dir's attr cache when its contents change
Any change to a directory's contents should cause its mtime and ctime to be
updated by the FUSE daemon.  Clear its attribute cache so we'll get the new
attributs the next time that they're needed.  This affects the following
VOPs: VOP_CREATE, VOP_LINK, VOP_MKDIR, VOP_MKNOD, VOP_REMOVE, VOP_RMDIR, and
VOP_SYMLINK

Reported by:	pjdfstest
Sponsored by:	The FreeBSD Foundation
2019-05-09 01:16:34 +00:00
Alan Somers
8e45ec4e64 fusefs: fix a permission handling bug during VOP_RENAME
If the file to be renamed is a directory and it's going to get a new parent,
then the user must have write permissions to that directory, because the
".." dirent must be changed.

Reported by:	pjdfstest
Sponsored by:	The FreeBSD Foundation
2019-05-08 22:28:13 +00:00
Alan Somers
d943c93e76 fusefs: allow non-owners to set timestamps to UTIME_NOW
utimensat should allow anybody with write access to set atime and mtime to
UTIME_NOW.

PR:		237181
Sponsored by:	The FreeBSD Foundation
2019-05-08 19:42:00 +00:00
Alan Somers
4ae3a56cb1 fusefs: updated cached attributes during VOP_LINK.
FUSE_LINK returns a new set of attributes.  fusefs should cache them just
like it does during other VOPs.  This is not only a matter of performance
but of correctness too; without caching the new attributes the vnode's nlink
value would be out-of-date.

Reported by:	pjdfstest
Sponsored by:	The FreeBSD Foundation
2019-05-08 18:12:38 +00:00
Alan Somers
a2bdd7379b fusefs: drop suid after a successful chown by a non-root user
Drop sgid too.  Also, drop them after a successful chgrp.

Reported by:	pjdfstest
Sponsored by:	The FreeBSD Foundation
2019-05-07 22:38:13 +00:00
Alan Somers
4e83d6555e fusefs: allow the null chown and null chgrp
Even an unprivileged user should be able to chown a file to its current
owner, or chgrp it to its current group.  Those are no-ops.

Reported by:	pjdfstest
Sponsored by:	The FreeBSD Foundation
2019-05-07 01:27:23 +00:00
Alan Somers
3fa127896b fusefs: allow ftruncate on files without write permission
ftruncate should succeed as long as the file descriptor is writable, even if
the file doesn't have write permission.  This is important when combined
with O_CREAT.

Reported by:	pjdfstest
Sponsored by:	The FreeBSD Foundation
2019-05-06 20:46:58 +00:00
Alan Somers
8cfb44315a fusefs: Fix another obscure permission handling bug
Don't allow unprivileged users to set SGID on files to whose group they
don't belong.  This is slightly different than what POSIX says we should do
(clear sgid on return from a successful chmod), but it matches what UFS
currently does.

Reported by:	pjdfstest
Sponsored by:	The FreeBSD Foundation
2019-05-06 16:54:35 +00:00
Alan Somers
a90e32de25 fusefs: clear SUID & SGID after a successful write by a non-owner
Reported by:	pjdfstest
Sponsored by:	The FreeBSD Foundation
2019-05-06 16:17:55 +00:00
Alan Somers
e5ff3a7e28 fusefs: only root may set the sticky bit on a non-directory
PR:		216391
Reported by:	pjdfstest
Sponsored by:	The FreeBSD Foundation
2019-05-04 16:27:58 +00:00
Alan Somers
61b0a927cb fusefs: use effective gid, not real gid, for FUSE operations
This is the gid used for stuff like setting the group of a newly created
file.

Reported by:	pjdfstest
Sponsored by:	The FreeBSD Foundation
2019-05-04 02:11:28 +00:00
Mark Johnston
8beadca53c Add a few regression tests for mlock(2).
These are intended to exercise some rarely executed code paths.

MFC after:	2 weeks
2019-05-01 15:28:23 +00:00
Alan Somers
474ba6fa3b fusefs: fix some permission checks with -o default_permissions
When mounted with -o default_permissions fusefs is supposed to validate all
permissions in the kernel, not the file system.  This commit fixes two
permissions that I had previously overlooked.

* Only root may chown a file
* Non-root users may only chgrp a file to a group to which they belong

PR:		216391
Sponsored by:	The FreeBSD Foundation
2019-05-01 00:00:49 +00:00
Alan Somers
ede571e40a fusefs: support unix-domain sockets
Also, fix the teardown of the Fifo.read_write test

Sponsored by:	The FreeBSD Foundation
2019-04-29 16:24:51 +00:00
Alan Somers
8acfadfa02 fusefs: remove an obsolete fifo test
This should've been part of r346868

Sponsored by:	The FreeBSD Foundation
2019-04-29 16:23:29 +00:00
Alan Somers
f9b0e30ba7 fusefs: FIFO support
Sponsored by:	The FreeBSD Foundation
2019-04-29 01:40:35 +00:00
Alan Somers
cf437e2aac fusefs: enable the Write.mmap test
This test had been disabled because it was designed to check protocol
7.9-specific functionality.  Enable it without the 7.9-specific bit.

Sponsored by:	The FreeBSD Foundation
2019-04-26 19:54:46 +00:00
Alan Somers
75d5cb29cb fusefs: fix cache invalidation error from r346162
An off-by-one error led to the last page of a write not being removed from
its object, even though that page's buffer was marked as invalid.

PR:		235774
Sponsored by:	The FreeBSD Foundation
2019-04-26 17:09:26 +00:00
Alan Somers
102c7ac083 fusefs: handle ENOSYS for FUSE_INTERRUPT
Though it's not documented, Linux will interpret a FUSE_INTERRUPT response
of ENOSYS as "the file system does not support FUSE_INTERRUPT".
Subsequently it will never send FUSE_INTERRUPT again to the same mount
point.  This change matches Linux's behavior.

PR:		346357
Sponsored by:	The FreeBSD Foundation
2019-04-24 17:30:50 +00:00
Alan Somers
9a17702912 fusefs: fix the FUSE_INTERRUPT tests when data_cache_mode==2
Replace most write operations with mkdir so they won't be affected by the
setting of vfs.fusefs.data_cache_mode.

Sponsored by:	The FreeBSD Foundation
2019-04-24 14:25:35 +00:00
Enji Cooper
351a56b116 Use range instead of xrange
`xrange` is a pre-python 2.x compatible idiom. Use `range` instead. The values
being iterated over are sufficiently small that using range on python 2.x won't
be a noticeable issue.

MFC after:	2 months
2019-04-24 05:52:24 +00:00
Enji Cooper
2a96ae15f0 Fix typo: Plen should be plen
MFC after:	1 month
MFC with:	r346617
Reported by:	pylint -E
2019-04-24 05:49:48 +00:00
Enji Cooper
56bf253633 Don't leak fd when manipulating the device via _getdev()
Close the file descriptor when done calling ioctl with a try-finally block so
it doesn't get leaked.

MFC after:	2 months
2019-04-24 05:47:09 +00:00
Enji Cooper
b106e0fccc Chase PEP-3110
Replace `except Environment, e:` with `except Environment as e` for
compatibility between python 2.x and python 3.x.

While here, fix a bad indentation change from r346620 by reindenting the code
properly.

MFC after:	2 months
2019-04-24 04:50:03 +00:00
Enji Cooper
ac65c82761 Reapply whitespace style changes from r346443 after recent changes to tests/sys/opencrypto
From r346443:
"""
Replace hard tabs with four-character indentations, per PEP8.

This is being done to separate stylistic changes from the tests from functional
ones, as I accidentally introduced a bug to the tests when I used four-space
indentation locally.

No functional change.
"""

MFC after:	2 months
Discussed with:	jhb
2019-04-24 04:40:24 +00:00
John Baldwin
151f0ca897 Test the AES-CCM test vectors from the NIST Known Answer Tests.
The CCM test vectors use a slightly different file format in that
there are global key-value pairs as well as section key-value pairs
that need to be used in each test.  In addition, the sections can set
multiple key-value pairs in the section name.  The CCM KAT parser
class is an iterator that returns a dictionary once per test where the
dictionary contains all of the relevant key-value pairs for a given
test (global, section name, section, test-specific).

Note that all of the CCM decrypt tests use nonce and tag lengths that
are not supported by OCF (OCF only supports a 12 byte nonce and 16
byte tag), so none of the decryption vectors are actually tested.

Reviewed by:	ngie
MFC after:	1 month
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D19978
2019-04-24 00:23:06 +00:00
John Baldwin
de0f7dca5e Run the plain SHA digest tests from NIST.
Pass in an explicit digest length to the Crypto constructor since it
was assuming only sessions with a MAC key would have a MAC.  Passing
an explicit size allows us to test the full digest in HMAC tests as
well.

Reviewed by:	cem
MFC after:	1 month
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D19884
2019-04-24 00:16:39 +00:00
John Baldwin
c091d0d95d Use more descriptive algorithm names in skip messages.
Reviewed by:	cem, ngie
MFC after:	1 month
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D19977
2019-04-24 00:14:37 +00:00
John Baldwin
aeb5c8e609 Skip tests with missing test vectors instead of failing.
This copes more gracefully when older version of the nist-kat package
are intalled that don't have newer test vectors such as CCM or plain
SHA.

If the nist-kat package is not installed at all, this still fails with
an error.

Reviewed by:	cem
MFC after:	1 month
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D20034
2019-04-24 00:10:21 +00:00
Alan Somers
77b8247874 Fix bug in vtruncbuf introduced by r346162
r346162 factored out v_inval_buf_range from vtruncbuf, but it made an error
in the interface between the two.  The result was a failure to remove
buffers past the first.  Surprisingly, I couldn't reproduce the failure with
file systems other than fuse.

Also, modify fusefs's truncate_discards_cached_data test to catch this bug.

PR:		346162
Sponsored by:	The FreeBSD Foundation
2019-04-23 22:22:46 +00:00
Olivier Cochard
9583ab8aa3 Skip test component_selection:run_latest_genid if gmirror/gnop GEOM classes
aren't available

PR:		237051
Reviewed by:	asomers, imp, ngie, emaste (IRC)
Approved by:	ngie
MFC after:	 1 month
Sponsored by:	Netflix
Differential Revision:	https://reviews.freebsd.org/D19958
2019-04-23 21:07:47 +00:00
Enji Cooper
616f60e87e Fix sys.kern.coredump_phnum_test.coredump_phnum on i386
The zero-padding when printing out the Size field is on 32-bit architectures is
5, not 15. Adjust the regular expression to work with both the 32-bit and
64-bit case.

MFC after:	1 week
Reviewed by:	lwhsu, markj
Approved by:	emaste (mentor, implicit)
Differential Revision: https://reviews.freebsd.org/D20005
2019-04-22 11:21:20 +00:00
Enji Cooper
03accca747 Revert r346443
My wide sweeping stylistic change (while well intended) is impeding others from
working on `tests/sys/opencrypto`.

The plan is to revert the change in ^/head, then reintroduce the changes after
the other changes get merged into ^/head .

Approved by:	emaste (mentor; implicit)
Requested by:	jhb
MFC after:	2 months
2019-04-20 16:37:28 +00:00
Enji Cooper
7bd1cac6c5 tests/sys/opencrypto: fix whitespace per PEP8
Replace hard tabs with four-character indentations, per PEP8.

This is being done to separate stylistic changes from the tests from functional
ones, as I accidentally introduced a bug to the tests when I used four-space
indentation locally.

No functional change.

MFC after:	2 months
Approved by:	emaste (mentor: implicit blanket approval for trivial fixes)
2019-04-20 15:43:28 +00:00
Li-Wen Hsu
369ee0905e Specify using Python2, these .py files have not been converted to use Python3
yet, but the default Python version in ports has been switched to 3.

Sponsored by:	The FreeBSD Foundation
2019-04-20 07:32:29 +00:00
John Baldwin
c87ada6a00 Test SHA2-224-HMAC now that OCF supports it.
Reviewed by:	cem
MFC after:	2 weeks
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D19882
2019-04-19 22:20:42 +00:00
John Baldwin
02babf9221 Sync cryptographic algorithm constants with current cryptodev.h.
Reviewed by:	cem
MFC after:	2 weeks
Sponsored by:	Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D19881
2019-04-19 21:58:51 +00:00
Alan Somers
268c28edbc fusefs: give priority to FUSE_INTERRUPT operations
When interrupting a FUSE operation, send the FUSE_INTERRUPT op to the daemon
ASAP, ahead of other unrelated operations.

PR:		236530
Sponsored by:	The FreeBSD Foundation
2019-04-19 21:50:23 +00:00
Alan Somers
f0f7fc1be4 fusefs: fix interrupting FUSE_SETXATTR
fusefs's VOP_SETEXTATTR calls uiomove(9) before blocking, so it can't be
restarted.  It must be interrupted instead.

PR:		236530
Sponsored by:	The FreeBSD Foundation
2019-04-19 20:31:12 +00:00
Alan Somers
3d070fdc76 fusefs: don't send FUSE_INTERRUPT for ops that are still in-kernel
If a pending FUSE operation hasn't yet been sent to the daemon, then there's
no reason to inform the daemon that it's been interrupted.  Instead, simply
remove it from the fuse message queue and set its status to EINTR or
ERESTART as appropriate.

PR:		346357
Sponsored by:	The FreeBSD Foundation
2019-04-19 15:05:32 +00:00
Alan Somers
a154214620 fusefs: improvements to interruptibility
* If a process receives a fatal signal while blocked on a fuse operation,
  return ASAP without waiting for the operation to complete.  But still send
  the FUSE_INTERRUPT op to the daemon.
* Plug memory leaks from r346339

Interruptibility is now fully functional, but it could be better:
* Operations that haven't been sent to the server yet should be aborted
  without sending FUSE_INTERRUPT.
* It would be great if write operations could be made restartable.
  That would require delaying uiomove until the last possible moment, which
  would be sometime during fuse_device_read.
* It would be nice if we didn't have to guess which EAGAIN responses were
  for FUSE_INTERRUPT operations.

PR:		236530
Sponsored by:	The FreeBSD Foundation
2019-04-18 19:16:34 +00:00
Kristof Provost
b20ff7b90a pf tests: Fail the test if we can't set the rules
The test should fail if pf rules can't be set. This is helpful both
while writing tests and to verify that pfctl works as expected.

MFC after:	1 week
Event:		Aberdeen hackathon 2019
2019-04-18 10:54:08 +00:00
Alan Somers
723c776829 fusefs: WIP making FUSE operations interruptible
The fuse protocol includes a FUSE_INTERRUPT operation that the client can
send to the server to indicate that it wants to abort an in-progress
operation.  It's required to interrupt any syscall that is blocking on a
fuse operation.

This commit adds basic FUSE_INTERRUPT support.  If a process receives any
signal while it's blocking on a FUSE operation, it will send a
FUSE_INTERRUPT and wait for the original operation to complete.  But there
is still much to do:

* The current code will leak memory if the server ignores FUSE_INTERRUPT,
  which many do.  It will also leak memory if the server completes the
  original operation before it receives the FUSE_INTERRUPT.
* An interrupted read(2) will incorrectly appear to be successful.
* fusefs should return immediately for fatal signals.
* Operations that haven't been sent to the server yet should be aborted
  without sending FUSE_INTERRUPT.
* Test coverage should be better.
* It would be great if write operations could be made restartable.
  That would require delaying uiomove until the last possible moment, which
  would be sometime during fuse_device_read.

PR:		236530
Sponsored by:	The FreeBSD Foundation
2019-04-17 23:32:38 +00:00
Kristof Provost
246e18b224 pf tests: Try to provoke the panic with invalid DIOCRSETTFLAGS
There was an issue with copyin() on DIOCRSETTFLAGS, which would panic if
pfrio_buffer was NULL.
Test for the issue fixed in r346319.

MFC after:	1 week
Event:		Aberdeen hackathon 2019
2019-04-17 16:45:35 +00:00
Vincenzo Maffione
4f6858e8a9 netmap: add test cases for multiple host rings
Extend the netmap unit tests with new test cases for the multiple-host-rings
feature introduced by r345269.

MFC after:	1 week
2019-04-13 12:50:47 +00:00
Alan Somers
f067b60946 fusefs: implement VOP_ADVLOCK
PR:		234581
Sponsored by:	The FreeBSD Foundation
2019-04-12 23:22:27 +00:00
Alan Somers
6af6fdcea7 fusefs: evict invalidated cache contents during write-through
fusefs's default cache mode is "writethrough", although it currently works
more like "write-around"; writes bypass the cache completely.  Since writes
bypass the cache, they were leaving stale previously-read data in the cache.
This commit invalidates that stale data.  It also adds a new global
v_inval_buf_range method, like vtruncbuf but for a range of a file.

PR:		235774
Reported by:	cem
Sponsored by:	The FreeBSD Foundation
2019-04-12 19:05:06 +00:00
Alan Somers
1f4a83f981 fusefs: Handle ENOSYS for all remaining opcodes
For many FUSE opcodes, an error of ENOSYS has special meaning.  fusefs
already handled some of those; this commit adds handling for the remainder:

* FUSE_FSYNC, FUSE_FSYNCDIR: ENOSYS means "success, and automatically return
  success without calling the daemon from now on"
* All extattr operations: ENOSYS means "fail EOPNOTSUPP, and automatically
  do it without calling the daemon from now on"

PR:		236557
Sponsored by:	The FreeBSD Foundation
2019-04-12 00:15:36 +00:00
Alan Somers
b349700a04 fusefs: add a test for setattr with UTIME_NOW
The test is disabled ATM; it requires protocol version 7.9.

PR:		237181
Sponsored by:	The FreeBSD Foundation
2019-04-11 23:15:37 +00:00
Alan Somers
c9c34c2057 fusefs: test that we reparent a vnode during rename
fusefs tracks each vnode's parent.  The rename code was already correctly
updating it.  Delete a comment that said otherwise, and add a regression
test for it.

Sponsored by:	The FreeBSD Foundation
2019-04-11 22:34:28 +00:00
Alan Somers
64f31d4f3b fusefs: fix a panic in a stale vnode situation
Don't panic if the server changes the file type of a file without us first
deleting it.  That could indicate a buggy server, but it could also be the
result of one of several race conditions.  Return EAGAIN as we do elsewhere.

Sponsored by:	The FreeBSD Foundation
2019-04-11 22:32:34 +00:00
Alan Somers
6124fd7106 fusefs: Finish supporting -o default_permissions
I got most of -o default_permissions working in r346088.  This commit adds
sticky bit checks.  One downside is that sometimes there will be an extra
FUSE_GETATTR call for the parent directory during unlink or rename.  But in
actual use I think those attributes will almost always be cached.

PR:		216391
Sponsored by:	The FreeBSD Foundation
2019-04-11 21:00:40 +00:00
Alan Somers
438b8a6fa2 fusefs: eliminate a superfluous FUSE_GETATTR from VOP_LOOKUP
fuse_vnop_lookup was using a FUSE_GETATTR operation when looking up "." and
"..", even though the only information it needed was the file type and file
size.  "." and ".." are obviously always going to be directories; there's no
need to double check.

Sponsored by:	The FreeBSD Foundation
2019-04-11 05:11:02 +00:00
Alan Somers
666f8543bb fusefs: various cleanups
* Eliminate fuse_access_param.  Whatever it was supposed to do, it seems
  like it was never complete.  The only real function it ever seems to have
  had was a minor performance optimization, which I've already eliminated.
* Make extended attribute operations obey the allow_other mount option.
* Allow unprivileged access to the SYSTEM extattr namespace when
  -o default_permissions is not in use.
* Disallow setextattr and deleteextattr on read-only mounts.
* Add tests for a few more error cases.

Sponsored by:	The FreeBSD Foundation
2019-04-10 21:10:21 +00:00
Alan Somers
ff4fbdf548 fusefs: WIP supporting -o default_permissions
Normally all permission checking is done in the fuse server.  But when -o
default_permissions is used, it should be done in the kernel instead.  This
commit adds appropriate permission checks through fusefs when -o
default_permissions is used.  However, sticky bit checks aren't working yet.
I'll handle those in a follow-up commit.

There are no checks for file flags, because those aren't supported by our
version of the FUSE protocol.  Nor is there any support for ACLs, though
that could be added if there were any demand.

PR:		216391
Reported by:	hiyorin@gmail.com
Sponsored by:	The FreeBSD Foundation
2019-04-10 17:31:00 +00:00
Alan Somers
df66475a5a fusefs: skip the Read.keep_cache test when cache is disabled
This should've been part of r345892

PR:		236560
Sponsored by:	The FreeBSD Foundation
2019-04-09 21:38:20 +00:00
Alan Somers
44f10c6e40 fusefs: cache negative lookups
The FUSE protocol includes a way for a server to tell the client that a
negative lookup response is cacheable for a certain amount of time.

PR:		236226
Sponsored by:	The FreeBSD Foundation
2019-04-09 21:22:02 +00:00
Enji Cooper
c9c9de9366 Polish netmap(4) testcases a bit
1. Not all kernels have netmap(4) support. Check for netmap(4) support before
   attempting to run the tests via the `PLAIN_REQUIRE_KERNEL_MODULE(..)` macro.
2. Libraries shouldn't be added to LDFLAGS; they should be added to LIBADD
   instead. This allows the build system to evaluate dependencies for sanity.
3. Sort some of the Makefile variables per bsd.README.

1., in particular, will resolve failures when running this testcase on kernels
lacking netmap(4) support, e.g., the i386 GENERIC kernels on ^/stable/11 and
^/stable/12.

PR:		237129
Reviewed by:	vmaffione
Approved by:	emaste (mentor)
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D19864
2019-04-09 17:52:11 +00:00
Alan Somers
ccb75e4939 fusefs: implement entry cache timeouts
Follow-up to r346046.  These two commits implement fuse cache timeouts for
both entries and attributes.  They also remove the vfs.fusefs.lookup_cache
enable sysctl, which is no longer needed now that cache timeouts are
honored.

PR:		235773
Sponsored by:	The FreeBSD Foundation
2019-04-09 17:23:34 +00:00
Enji Cooper
724ec8989b Fix geli device cleanup
Final cleanup routines shouldn't be called from testcases; it should be called
from the testcase cleanup routine.

Furthermore, `geli_test_cleanup` should take care of cleaning up geli providers
and the memory disks used for the geli providers. `geli_test_cleanup` will always
be executed whereas the equivalent logic in `geli_test_body`, may not have been
executed if the test failed prior to the logic being run.

Prior to this change, the test case was trying to clean up `$md` twice: once in
at the end of the test case body function, and the other in the cleanup function.
The cleanup function logic was failing because there wasn't anything to clean up
in the cleanup function and the errors weren't being ignored.

This fixes FreeBSD test suite runs after r345864.

PR:		237128
Reviewed by:	asomers, pjd
Approved by:	emaste (mentor)
MFC with:	r345864
Differential Revision:	https://reviews.freebsd.org/D19854
2019-04-09 16:20:36 +00:00
Alan Somers
3f2c630c74 fusefs: implement attribute cache timeouts
The FUSE protocol allows the server to specify the timeout period for the
client's attribute and entry caches.  This commit implements the timeout
period for the attribute cache.  The entry cache's timeout period is
currently disabled because it panics, and is guarded by the
vfs.fusefs.lookup_cache_expire sysctl.

PR:		235773
Reported by:	cem
Sponsored by:	The FreeBSD Foundation
2019-04-09 00:47:38 +00:00
Alan Somers
cad677915f fusefs: cache file attributes
FUSE_LOOKUP, FUSE_GETATTR, FUSE_SETATTR, FUSE_MKDIR, FUSE_LINK,
FUSE_SYMLINK, FUSE_MKNOD, and FUSE_CREATE all return file attributes with a
cache validity period.  fusefs will now cache the attributes, if the server
returns a non-zero cache validity period.

This change does _not_ implement finite attr cache timeouts.  That will
follow as part of PR 235773.

PR:		235775
Reported by:	cem
Sponsored by:	The FreeBSD Foundation
2019-04-08 18:45:41 +00:00
Mark Johnston
128c9bc05b Set the p_oppid field of orphans when exiting.
Such processes will be reparented to the reaper when the current
parent is done with them (i.e., ptrace detached), so p_oppid must be
updated accordingly.

Add a regression test to exercise this code path.  Previously it
would not be possible to reap an orphan with a stale oppid.

Reviewed by:	kib, mjg
Tested by:	pho
MFC after:	2 weeks
Sponsored by:	The FreeBSD Foundation
Differential Revision:	https://reviews.freebsd.org/D19825
2019-04-07 14:26:14 +00:00
Alan Somers
caf5f57d2d fusefs: implement VOP_ACCESS
VOP_ACCESS was never fully implemented in fusefs.  This change:
* Removes the FACCESS_DO_ACCESS flag, which pretty much disabled the whole
  vop.
* Removes a quixotic special case for VEXEC on regular files.  I don't know
  why that was in there.
* Removes another confusing special case for VADMIN.
* Removes the FACCESS_NOCHECKSPY flag.  It seemed to be a performance
  optimization, but I'm unconvinced that it was a net positive.
* Updates test cases.

This change does NOT implement -o default_permissions.  That will be handled
separately.

PR:		236291
Sponsored by:	The FreeBSD Foundation
2019-04-05 18:37:48 +00:00
Alan Somers
efa23d9784 fusefs: enforce -onoallow_other even beneath the mountpoint
When -o allow_other is not in use, fusefs is supposed to prevent access to
the filesystem by any user other than the one who owns the daemon.  Our
fusefs implementation was only enforcing that restriction at the mountpoint
itself.  That was usually good enough because lookup usually descends from
the mountpoint.  However, there are cases when it doesn't, such as when
using openat relative to a file beneath the mountpoint.

PR:		237052
Sponsored by:	The FreeBSD Foundation
2019-04-05 17:21:23 +00:00
Alan Somers
140bb4927a fusefs: correctly return EROFS from VOP_ACCESS
Sponsored by:	The FreeBSD Foundation
2019-04-05 15:33:43 +00:00
Alan Somers
46c37cd0d7 fusefs: reenable some fsyncdir tests
These tests were actually fixed by r345398, r345390 and r345392, but I
neglected to reenable them.  Too bad googletest doesn't have the notion of
an Expected Failure like ATF does.

PR:		236474, 236473
Sponsored by:	The FreeBSD Foundation
2019-04-05 15:04:25 +00:00
Alan Somers
a7e81cb3db fusefs: properly handle FOPEN_KEEP_CACHE
If a fuse file system returne FOPEN_KEEP_CACHE in the open or create
response, then the client is supposed to _not_ clear its caches for that
file.  I don't know why clearing the caches would be the default given that
there's a separate flag to bypass the cache altogether, but that's the way
it is.  fusefs(5) will now honor this flag.

Our behavior is slightly different than Linux's because we reuse file
handles.  That means that open(2) wont't clear the cache if there's a
reusable file handle, even if the file server wouldn't have sent
FOPEN_KEEP_CACHE had we opened a new file handle like Linux does.

PR:		236560
Sponsored by:	The FreeBSD Foundation
2019-04-04 20:30:14 +00:00
Alan Somers
9a696dc6bb MFHead@r345880 2019-04-04 18:26:32 +00:00
Alan Somers
12292a99ac fusefs: correctly handle short writes
If a FUSE daemon returns FOPEN_DIRECT_IO when a file is opened, then it's
allowed to write less data than was requested during a FUSE_WRITE operation
on that file handle.  fusefs should simply return a short write to userland.

The old code attempted to resend the unsent data.  Not only was that
incorrect behavior, but it did it in an ineffective way, by attempting to
"rewind" the uio and uiomove the unsent data again.

This commit correctly handles short writes by returning directly to
userland if FOPEN_DIRECT_IO was set.  If it wasn't set (making the short
write technically a protocol violation), then we resend the unsent data.
But instead of rewinding the uio, just resend the data that's already in the
kernel.

That necessitated a few changes to fuse_ipc.c to reduce the amount of bzero
activity.  fusefs may be marginally faster as a result.

PR:		236381
Sponsored by:	The FreeBSD Foundation
2019-04-04 16:51:34 +00:00
Pawel Jakub Dawidek
d9eb18ace9 Implement tests for online expansion:
- init, init -R
- onetime, onetime -R
- 512 and 4k sectors
- encryption only
- encryption and authentication
- configure -r/-R for detached providers
- configure -r/-R for attached providers
- all keys allocated (10, 20 and 30MB provider sizes)
- keys allocated on demand (10, 20 and 30PB provider sizes)
- reading and writing to provider after expansion (10-30MB only)
- checking if metadata in old location is cleared.

Obtained from:	Fudo Security
2019-04-04 00:05:36 +00:00
Pawel Jakub Dawidek
1e560a0ca4 Update configure tests after addition of the online expansion.
Obtained from:	Fudo Security
2019-04-03 23:58:58 +00:00
Alan Somers
35cf0e7e56 fusefs: fix a panic in VOP_READDIR
The original fusefs import, r238402, contained a bug in fuse_vnop_close that
could close a directory's file handle while there were still other open file
descriptors.  The code looks deliberate, but there is no explanation for it.
This necessitated a workaround in fuse_vnop_readdir that would open a new
file handle if, "for some mysterious reason", that vnode didn't have any
open file handles.  r345781 had the effect of causing the workaround to
panic, making the problem more visible.

This commit removes the workaround and the original bug, which also fixes
the panic.

Sponsored by:	The FreeBSD Foundation
2019-04-03 20:57:43 +00:00
Alan Somers
9f10f423a9 fusefs: send FUSE_FLUSH during VOP_CLOSE
The FUSE protocol says that FUSE_FLUSH should be send every time a file
descriptor is closed.  That's not quite possible in FreeBSD because multiple
file descriptors can share a single struct file, and closef doesn't call
fo_close until the last close.  However, we can still send FUSE_FLUSH on
every VOP_CLOSE, which is probably good enough.

There are two purposes for FUSE_FLUSH.  One is to allow file systems to
return EIO if they have an error when writing data that's cached
server-side.  The other is to release POSIX file locks (which fusefs(5) does
not yet support).

PR:		236405, 236327
Sponsored by:	The FreeBSD Foundation
2019-04-03 19:59:45 +00:00
Alan Somers
e312493b37 fusefs: during ftruncate, discard cached data past truncation point
During truncate, fusefs was discarding entire cached blocks, but it wasn't
zeroing out the unused portion of a final partial block.  This resulted in
reads returning stale data.

PR:		233783
Reported by:	fsx
Sponsored by:	The FreeBSD Foundation
2019-04-03 02:29:56 +00:00
Alan Somers
4eb8481630 fusefs: check return value of wait(2) in fork tests
Reported by:	ngie
Sponsored by:	The FreeBSD Foundation
2019-04-02 18:44:01 +00:00