In python 3, the default encoding was switched from ascii character sets to
unicode character sets in order to support internationalization by default.
Some interfaces, like ioctls and packets, however, specify data in terms of
non-unicode encodings formats, either in host endian (`fcntl.ioctl`) or
network endian (`dpkt`) byte order/format.
This change alters assumptions made by previous code where it was all
data objects were assumed to be basestrings, when they should have been
treated as byte arrays. In order to achieve this the following are done:
* str objects with encodings needing to be encoded as ascii byte arrays are
done so via `.encode("ascii")`. In order for this to work on python 3 in a
type agnostic way (as it anecdotally varied depending on the caller), call
`.encode("ascii")` only on str objects with python 3 to cast them to ascii
byte arrays in a helper function name `str_to_ascii(..)`.
* `dpkt.Packet` objects needing to be passed in to `fcntl.ioctl(..)` are done
so by casting them to byte arrays via `bytes()`, which calls
`dpkt.Packet__str__` under the covers and does the necessary str to byte array
conversion needed for the `dpkt` APIs and `struct` module.
In order to accomodate this change, apply the necessary typecasting for the
byte array literal in order to search `fop.name` for nul bytes.
This resolves all remaining python 2.x and python 3.x compatibility issues on
amd64. More work needs to be done for the tests to function with i386, in
general (this is a legacy issue).
PR: 237403
MFC after: 1 week
Tested with: python 2.7.16 (amd64), python 3.6.8 (amd64)
Even though some python styles suggest there should be multiple newlines between
methods/classes, for consistency with the surrounding code, it's best to be
consistent by having merely one newline between each functional block.
MFC after: 1 week
Make `KAT(CCM)?Parser` into a context suite-capable object by implementing
`__enter__` and `__exit__` methods which manage opening up the file descriptors
and closing them on context exit. This implementation was decided over adding
destructor logic to a `__del__` method, as there are a number of issues around
object lifetimes when dealing with threading cleanup, atexit handlers, and a
number of other less obvious edgecases. Plus, the architected solution is more
pythonic and clean.
Complete the iterator implementation by implementing a `__next__` method for
both classes which handles iterating over the data using a generator pattern,
and by changing `__iter__` to return the object instead of the data which it
would iterate over. Alias the `__next__` method to `next` when working with
python 2.x in order to maintain functional compatibility between the two major
versions.
As part of this work and to ensure readability, push the initialization of the
parser objects up one layer and pass it down to a helper function. This could
have been done via a decorator, but I was trying to keep it simple for other
developers to make it easier to modify in the future.
This fixes ResourceWarnings with python 3.
PR: 237403
MFC after: 1 week
Tested with: python 2.7.16 (amd64), python 3.6.8 (amd64)
In version 3.2+, `array.array(..).tostring()` was renamed to
`array.array(..).tobytes()`. Conditionally call `array.array(..).tobytes()` if
the python version is 3.2+.
PR: 237403
MFC after: 1 week
Replace uses of `foo.encode("hex")` with `binascii.hexlify(foo)` for forwards
compatibility between python 2.x and python 3.
PR: 237403
MFC after: 1 week
Python 3 no longer doesn't support encoding/decoding hexadecimal numbers using
the `str.format` method. The backwards compatible new method (using the
binascii module/methods) is a comparable means of converting to/from
hexadecimal format.
In short, the functional change is the following:
* `foo.decode('hex')` -> `binascii.unhexlify(foo)`
* `foo.encode('hex')` -> `binascii.hexlify(foo)`
While here, move the dpkt import in `cryptodev.py` down per PEP8, so it comes
after the standard library provided imports.
PR: 237403
MFC after: 1 week
If a user sets both atime and mtime to UTIME_NOW when calling a syscall like
utimensat(2), allow the server to choose what "now" means. Due to the
design of FreeBSD's VFS, it's not possible to do this for just one of atime
or mtime; it's all or none.
PR: 237181
Sponsored by: The FreeBSD Foundation
If the server sets fuse_attr.blksize to a nonzero value in the response to
FUSE_GETATTR, then the client should use that as the value for
stat.st_blksize .
Sponsored by: The FreeBSD Foundation
This commit upgrades the FUSE API to protocol 7.9 and adds unit tests for
backwards compatibility with servers built for version 7.8. It doesn't
implement any of 7.9's new features yet.
Sponsored by: The FreeBSD Foundation
As of r347410 IPSec is no longer built into GENERIC. The ipsec.ko module must
be loaded before we can execute the IPSec tests.
Check this, and skip the tests if IPSec is not available.
When using poll, kevent, or select there was a race window during which it
would be impossible to shut down the daemon. The problem was that poll,
kevent, and select don't return when the file descriptor gets closed (or
maybe it was that the file descriptor got closed before those syscalls were
entered?). The solution is to impose a timeout on those syscalls, and check
m_quit after they time out.
Sponsored by: The FreeBSD Foundation
Just like /dev/devctl, /dev/fuse will now report the number of operations
available for immediate read in the kevent.data field during kevent(2).
Sponsored by: The FreeBSD Foundation
/dev/fuse was already pollable with poll and select. Add support for
kqueue, too. And add tests for polling with poll, select, and kqueue.
Sponsored by: The FreeBSD Foundation
* In the fatal_signal test, wait for the daemon to receive FUSE_INTERRUPT
before exiting.
* Explicitly disable restarting syscalls after SIGUSR2. This fixes
intermittency in the priority test. I don't know why, but sometimes that
test's mkdir would be restarted, and sometimes it would return EINTR.
ERESTART should be the default.
* Remove a useless copy/pasted sleep in the priority test.
Sponsored by: The FreeBSD Foundation
If the daemon dies, return ENOTCONN for all operations that have already
been sent to the daemon, as well as any new ones.
Sponsored by: The FreeBSD Foundation
When a FUSE daemon dies or closes /dev/fuse, all of that daemon's pending
requests must be terminated. Previously that was done in /dev/fuse's
.d_close method. However, d_close only gets called on the *last* close of
the device. That means that if multiple daemons were running concurrently,
all but the last daemon to close would leave their I/O hanging around. The
problem was easily visible just by running "kyua -v parallelism=2 test" in
fusefs's test directory.
Fix this bug by terminating a daemon's pending I/O during /dev/fuse's
cdvpriv dtor method instead. That method runs on every close of a file.
Also, fix some potential races in the tests:
* Clear SA_RESTART when registering the daemon's signal handler so read(2)
will return EINTR.
* Wait for the daemon to die before unmounting the mountpoint, so we won't
see an unwanted FUSE_DESTROY operation in the mock file system.
Sponsored by: The FreeBSD Foundation
* Convert from plain to TAP for slightly improved introspection when skipping
the tests due to requirements not being met.
* Test for the net/py-dpkt (origin) package being required when running the
tests, instead of relying on a copy of the dpkt.py module from 2014. This
enables the tests to work with py3. Subsequently, remove
`tests/sys/opencrypto/dpkt.py(c)?` via `make delete-old`.
* Parameterize out `python2` as `$PYTHON`.
PR: 237403
MFC after: 1 week
Some fusefs tests must sleep because they deliberately trigger a race, or
because they're testing the cache timeout functionality. Consolidate the
sleep interval in a single place so it will be easy to adjust. Shorten it
from either 500ms or 250ms to 100ms. From experiment I find that 10ms works
every time, so 100ms should be fairly safe.
Sponsored by: The FreeBSD Foundation
Replace some sleeps with semaphore operations. Not all sleeps can be
replaced, though. Some are trying to lose a race.
Sponsored by: The FreeBSD Foundation
libfuse expects sockets to be created with FUSE_MKNOD, not FUSE_CREATE,
because that's how Linux does it. My first attempt at creating sockets
(r346894) used FUSE_CREATE because FreeBSD uses VOP_CREATE for this purpose.
There are no backwards-compatibility concerns with this change, because
socket support hasn't yet been merged to head.
Sponsored by: The FreeBSD Foundation
Any change to a directory's contents should cause its mtime and ctime to be
updated by the FUSE daemon. Clear its attribute cache so we'll get the new
attributs the next time that they're needed. This affects the following
VOPs: VOP_CREATE, VOP_LINK, VOP_MKDIR, VOP_MKNOD, VOP_REMOVE, VOP_RMDIR, and
VOP_SYMLINK
Reported by: pjdfstest
Sponsored by: The FreeBSD Foundation
If the file to be renamed is a directory and it's going to get a new parent,
then the user must have write permissions to that directory, because the
".." dirent must be changed.
Reported by: pjdfstest
Sponsored by: The FreeBSD Foundation
FUSE_LINK returns a new set of attributes. fusefs should cache them just
like it does during other VOPs. This is not only a matter of performance
but of correctness too; without caching the new attributes the vnode's nlink
value would be out-of-date.
Reported by: pjdfstest
Sponsored by: The FreeBSD Foundation
Even an unprivileged user should be able to chown a file to its current
owner, or chgrp it to its current group. Those are no-ops.
Reported by: pjdfstest
Sponsored by: The FreeBSD Foundation
ftruncate should succeed as long as the file descriptor is writable, even if
the file doesn't have write permission. This is important when combined
with O_CREAT.
Reported by: pjdfstest
Sponsored by: The FreeBSD Foundation
Don't allow unprivileged users to set SGID on files to whose group they
don't belong. This is slightly different than what POSIX says we should do
(clear sgid on return from a successful chmod), but it matches what UFS
currently does.
Reported by: pjdfstest
Sponsored by: The FreeBSD Foundation
When mounted with -o default_permissions fusefs is supposed to validate all
permissions in the kernel, not the file system. This commit fixes two
permissions that I had previously overlooked.
* Only root may chown a file
* Non-root users may only chgrp a file to a group to which they belong
PR: 216391
Sponsored by: The FreeBSD Foundation
This test had been disabled because it was designed to check protocol
7.9-specific functionality. Enable it without the 7.9-specific bit.
Sponsored by: The FreeBSD Foundation
An off-by-one error led to the last page of a write not being removed from
its object, even though that page's buffer was marked as invalid.
PR: 235774
Sponsored by: The FreeBSD Foundation
Though it's not documented, Linux will interpret a FUSE_INTERRUPT response
of ENOSYS as "the file system does not support FUSE_INTERRUPT".
Subsequently it will never send FUSE_INTERRUPT again to the same mount
point. This change matches Linux's behavior.
PR: 346357
Sponsored by: The FreeBSD Foundation
`xrange` is a pre-python 2.x compatible idiom. Use `range` instead. The values
being iterated over are sufficiently small that using range on python 2.x won't
be a noticeable issue.
MFC after: 2 months
Replace `except Environment, e:` with `except Environment as e` for
compatibility between python 2.x and python 3.x.
While here, fix a bad indentation change from r346620 by reindenting the code
properly.
MFC after: 2 months
From r346443:
"""
Replace hard tabs with four-character indentations, per PEP8.
This is being done to separate stylistic changes from the tests from functional
ones, as I accidentally introduced a bug to the tests when I used four-space
indentation locally.
No functional change.
"""
MFC after: 2 months
Discussed with: jhb
The CCM test vectors use a slightly different file format in that
there are global key-value pairs as well as section key-value pairs
that need to be used in each test. In addition, the sections can set
multiple key-value pairs in the section name. The CCM KAT parser
class is an iterator that returns a dictionary once per test where the
dictionary contains all of the relevant key-value pairs for a given
test (global, section name, section, test-specific).
Note that all of the CCM decrypt tests use nonce and tag lengths that
are not supported by OCF (OCF only supports a 12 byte nonce and 16
byte tag), so none of the decryption vectors are actually tested.
Reviewed by: ngie
MFC after: 1 month
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D19978
Pass in an explicit digest length to the Crypto constructor since it
was assuming only sessions with a MAC key would have a MAC. Passing
an explicit size allows us to test the full digest in HMAC tests as
well.
Reviewed by: cem
MFC after: 1 month
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D19884
This copes more gracefully when older version of the nist-kat package
are intalled that don't have newer test vectors such as CCM or plain
SHA.
If the nist-kat package is not installed at all, this still fails with
an error.
Reviewed by: cem
MFC after: 1 month
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D20034
r346162 factored out v_inval_buf_range from vtruncbuf, but it made an error
in the interface between the two. The result was a failure to remove
buffers past the first. Surprisingly, I couldn't reproduce the failure with
file systems other than fuse.
Also, modify fusefs's truncate_discards_cached_data test to catch this bug.
PR: 346162
Sponsored by: The FreeBSD Foundation
The zero-padding when printing out the Size field is on 32-bit architectures is
5, not 15. Adjust the regular expression to work with both the 32-bit and
64-bit case.
MFC after: 1 week
Reviewed by: lwhsu, markj
Approved by: emaste (mentor, implicit)
Differential Revision: https://reviews.freebsd.org/D20005
My wide sweeping stylistic change (while well intended) is impeding others from
working on `tests/sys/opencrypto`.
The plan is to revert the change in ^/head, then reintroduce the changes after
the other changes get merged into ^/head .
Approved by: emaste (mentor; implicit)
Requested by: jhb
MFC after: 2 months
Replace hard tabs with four-character indentations, per PEP8.
This is being done to separate stylistic changes from the tests from functional
ones, as I accidentally introduced a bug to the tests when I used four-space
indentation locally.
No functional change.
MFC after: 2 months
Approved by: emaste (mentor: implicit blanket approval for trivial fixes)
When interrupting a FUSE operation, send the FUSE_INTERRUPT op to the daemon
ASAP, ahead of other unrelated operations.
PR: 236530
Sponsored by: The FreeBSD Foundation
fusefs's VOP_SETEXTATTR calls uiomove(9) before blocking, so it can't be
restarted. It must be interrupted instead.
PR: 236530
Sponsored by: The FreeBSD Foundation
If a pending FUSE operation hasn't yet been sent to the daemon, then there's
no reason to inform the daemon that it's been interrupted. Instead, simply
remove it from the fuse message queue and set its status to EINTR or
ERESTART as appropriate.
PR: 346357
Sponsored by: The FreeBSD Foundation
* If a process receives a fatal signal while blocked on a fuse operation,
return ASAP without waiting for the operation to complete. But still send
the FUSE_INTERRUPT op to the daemon.
* Plug memory leaks from r346339
Interruptibility is now fully functional, but it could be better:
* Operations that haven't been sent to the server yet should be aborted
without sending FUSE_INTERRUPT.
* It would be great if write operations could be made restartable.
That would require delaying uiomove until the last possible moment, which
would be sometime during fuse_device_read.
* It would be nice if we didn't have to guess which EAGAIN responses were
for FUSE_INTERRUPT operations.
PR: 236530
Sponsored by: The FreeBSD Foundation
The test should fail if pf rules can't be set. This is helpful both
while writing tests and to verify that pfctl works as expected.
MFC after: 1 week
Event: Aberdeen hackathon 2019
The fuse protocol includes a FUSE_INTERRUPT operation that the client can
send to the server to indicate that it wants to abort an in-progress
operation. It's required to interrupt any syscall that is blocking on a
fuse operation.
This commit adds basic FUSE_INTERRUPT support. If a process receives any
signal while it's blocking on a FUSE operation, it will send a
FUSE_INTERRUPT and wait for the original operation to complete. But there
is still much to do:
* The current code will leak memory if the server ignores FUSE_INTERRUPT,
which many do. It will also leak memory if the server completes the
original operation before it receives the FUSE_INTERRUPT.
* An interrupted read(2) will incorrectly appear to be successful.
* fusefs should return immediately for fatal signals.
* Operations that haven't been sent to the server yet should be aborted
without sending FUSE_INTERRUPT.
* Test coverage should be better.
* It would be great if write operations could be made restartable.
That would require delaying uiomove until the last possible moment, which
would be sometime during fuse_device_read.
PR: 236530
Sponsored by: The FreeBSD Foundation
There was an issue with copyin() on DIOCRSETTFLAGS, which would panic if
pfrio_buffer was NULL.
Test for the issue fixed in r346319.
MFC after: 1 week
Event: Aberdeen hackathon 2019
fusefs's default cache mode is "writethrough", although it currently works
more like "write-around"; writes bypass the cache completely. Since writes
bypass the cache, they were leaving stale previously-read data in the cache.
This commit invalidates that stale data. It also adds a new global
v_inval_buf_range method, like vtruncbuf but for a range of a file.
PR: 235774
Reported by: cem
Sponsored by: The FreeBSD Foundation
For many FUSE opcodes, an error of ENOSYS has special meaning. fusefs
already handled some of those; this commit adds handling for the remainder:
* FUSE_FSYNC, FUSE_FSYNCDIR: ENOSYS means "success, and automatically return
success without calling the daemon from now on"
* All extattr operations: ENOSYS means "fail EOPNOTSUPP, and automatically
do it without calling the daemon from now on"
PR: 236557
Sponsored by: The FreeBSD Foundation
fusefs tracks each vnode's parent. The rename code was already correctly
updating it. Delete a comment that said otherwise, and add a regression
test for it.
Sponsored by: The FreeBSD Foundation
Don't panic if the server changes the file type of a file without us first
deleting it. That could indicate a buggy server, but it could also be the
result of one of several race conditions. Return EAGAIN as we do elsewhere.
Sponsored by: The FreeBSD Foundation
I got most of -o default_permissions working in r346088. This commit adds
sticky bit checks. One downside is that sometimes there will be an extra
FUSE_GETATTR call for the parent directory during unlink or rename. But in
actual use I think those attributes will almost always be cached.
PR: 216391
Sponsored by: The FreeBSD Foundation
fuse_vnop_lookup was using a FUSE_GETATTR operation when looking up "." and
"..", even though the only information it needed was the file type and file
size. "." and ".." are obviously always going to be directories; there's no
need to double check.
Sponsored by: The FreeBSD Foundation
* Eliminate fuse_access_param. Whatever it was supposed to do, it seems
like it was never complete. The only real function it ever seems to have
had was a minor performance optimization, which I've already eliminated.
* Make extended attribute operations obey the allow_other mount option.
* Allow unprivileged access to the SYSTEM extattr namespace when
-o default_permissions is not in use.
* Disallow setextattr and deleteextattr on read-only mounts.
* Add tests for a few more error cases.
Sponsored by: The FreeBSD Foundation
Normally all permission checking is done in the fuse server. But when -o
default_permissions is used, it should be done in the kernel instead. This
commit adds appropriate permission checks through fusefs when -o
default_permissions is used. However, sticky bit checks aren't working yet.
I'll handle those in a follow-up commit.
There are no checks for file flags, because those aren't supported by our
version of the FUSE protocol. Nor is there any support for ACLs, though
that could be added if there were any demand.
PR: 216391
Reported by: hiyorin@gmail.com
Sponsored by: The FreeBSD Foundation
The FUSE protocol includes a way for a server to tell the client that a
negative lookup response is cacheable for a certain amount of time.
PR: 236226
Sponsored by: The FreeBSD Foundation
1. Not all kernels have netmap(4) support. Check for netmap(4) support before
attempting to run the tests via the `PLAIN_REQUIRE_KERNEL_MODULE(..)` macro.
2. Libraries shouldn't be added to LDFLAGS; they should be added to LIBADD
instead. This allows the build system to evaluate dependencies for sanity.
3. Sort some of the Makefile variables per bsd.README.
1., in particular, will resolve failures when running this testcase on kernels
lacking netmap(4) support, e.g., the i386 GENERIC kernels on ^/stable/11 and
^/stable/12.
PR: 237129
Reviewed by: vmaffione
Approved by: emaste (mentor)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D19864
Follow-up to r346046. These two commits implement fuse cache timeouts for
both entries and attributes. They also remove the vfs.fusefs.lookup_cache
enable sysctl, which is no longer needed now that cache timeouts are
honored.
PR: 235773
Sponsored by: The FreeBSD Foundation
Final cleanup routines shouldn't be called from testcases; it should be called
from the testcase cleanup routine.
Furthermore, `geli_test_cleanup` should take care of cleaning up geli providers
and the memory disks used for the geli providers. `geli_test_cleanup` will always
be executed whereas the equivalent logic in `geli_test_body`, may not have been
executed if the test failed prior to the logic being run.
Prior to this change, the test case was trying to clean up `$md` twice: once in
at the end of the test case body function, and the other in the cleanup function.
The cleanup function logic was failing because there wasn't anything to clean up
in the cleanup function and the errors weren't being ignored.
This fixes FreeBSD test suite runs after r345864.
PR: 237128
Reviewed by: asomers, pjd
Approved by: emaste (mentor)
MFC with: r345864
Differential Revision: https://reviews.freebsd.org/D19854
The FUSE protocol allows the server to specify the timeout period for the
client's attribute and entry caches. This commit implements the timeout
period for the attribute cache. The entry cache's timeout period is
currently disabled because it panics, and is guarded by the
vfs.fusefs.lookup_cache_expire sysctl.
PR: 235773
Reported by: cem
Sponsored by: The FreeBSD Foundation
FUSE_LOOKUP, FUSE_GETATTR, FUSE_SETATTR, FUSE_MKDIR, FUSE_LINK,
FUSE_SYMLINK, FUSE_MKNOD, and FUSE_CREATE all return file attributes with a
cache validity period. fusefs will now cache the attributes, if the server
returns a non-zero cache validity period.
This change does _not_ implement finite attr cache timeouts. That will
follow as part of PR 235773.
PR: 235775
Reported by: cem
Sponsored by: The FreeBSD Foundation
Such processes will be reparented to the reaper when the current
parent is done with them (i.e., ptrace detached), so p_oppid must be
updated accordingly.
Add a regression test to exercise this code path. Previously it
would not be possible to reap an orphan with a stale oppid.
Reviewed by: kib, mjg
Tested by: pho
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D19825
VOP_ACCESS was never fully implemented in fusefs. This change:
* Removes the FACCESS_DO_ACCESS flag, which pretty much disabled the whole
vop.
* Removes a quixotic special case for VEXEC on regular files. I don't know
why that was in there.
* Removes another confusing special case for VADMIN.
* Removes the FACCESS_NOCHECKSPY flag. It seemed to be a performance
optimization, but I'm unconvinced that it was a net positive.
* Updates test cases.
This change does NOT implement -o default_permissions. That will be handled
separately.
PR: 236291
Sponsored by: The FreeBSD Foundation
When -o allow_other is not in use, fusefs is supposed to prevent access to
the filesystem by any user other than the one who owns the daemon. Our
fusefs implementation was only enforcing that restriction at the mountpoint
itself. That was usually good enough because lookup usually descends from
the mountpoint. However, there are cases when it doesn't, such as when
using openat relative to a file beneath the mountpoint.
PR: 237052
Sponsored by: The FreeBSD Foundation
These tests were actually fixed by r345398, r345390 and r345392, but I
neglected to reenable them. Too bad googletest doesn't have the notion of
an Expected Failure like ATF does.
PR: 236474, 236473
Sponsored by: The FreeBSD Foundation
If a fuse file system returne FOPEN_KEEP_CACHE in the open or create
response, then the client is supposed to _not_ clear its caches for that
file. I don't know why clearing the caches would be the default given that
there's a separate flag to bypass the cache altogether, but that's the way
it is. fusefs(5) will now honor this flag.
Our behavior is slightly different than Linux's because we reuse file
handles. That means that open(2) wont't clear the cache if there's a
reusable file handle, even if the file server wouldn't have sent
FOPEN_KEEP_CACHE had we opened a new file handle like Linux does.
PR: 236560
Sponsored by: The FreeBSD Foundation
If a FUSE daemon returns FOPEN_DIRECT_IO when a file is opened, then it's
allowed to write less data than was requested during a FUSE_WRITE operation
on that file handle. fusefs should simply return a short write to userland.
The old code attempted to resend the unsent data. Not only was that
incorrect behavior, but it did it in an ineffective way, by attempting to
"rewind" the uio and uiomove the unsent data again.
This commit correctly handles short writes by returning directly to
userland if FOPEN_DIRECT_IO was set. If it wasn't set (making the short
write technically a protocol violation), then we resend the unsent data.
But instead of rewinding the uio, just resend the data that's already in the
kernel.
That necessitated a few changes to fuse_ipc.c to reduce the amount of bzero
activity. fusefs may be marginally faster as a result.
PR: 236381
Sponsored by: The FreeBSD Foundation
- init, init -R
- onetime, onetime -R
- 512 and 4k sectors
- encryption only
- encryption and authentication
- configure -r/-R for detached providers
- configure -r/-R for attached providers
- all keys allocated (10, 20 and 30MB provider sizes)
- keys allocated on demand (10, 20 and 30PB provider sizes)
- reading and writing to provider after expansion (10-30MB only)
- checking if metadata in old location is cleared.
Obtained from: Fudo Security
The original fusefs import, r238402, contained a bug in fuse_vnop_close that
could close a directory's file handle while there were still other open file
descriptors. The code looks deliberate, but there is no explanation for it.
This necessitated a workaround in fuse_vnop_readdir that would open a new
file handle if, "for some mysterious reason", that vnode didn't have any
open file handles. r345781 had the effect of causing the workaround to
panic, making the problem more visible.
This commit removes the workaround and the original bug, which also fixes
the panic.
Sponsored by: The FreeBSD Foundation
The FUSE protocol says that FUSE_FLUSH should be send every time a file
descriptor is closed. That's not quite possible in FreeBSD because multiple
file descriptors can share a single struct file, and closef doesn't call
fo_close until the last close. However, we can still send FUSE_FLUSH on
every VOP_CLOSE, which is probably good enough.
There are two purposes for FUSE_FLUSH. One is to allow file systems to
return EIO if they have an error when writing data that's cached
server-side. The other is to release POSIX file locks (which fusefs(5) does
not yet support).
PR: 236405, 236327
Sponsored by: The FreeBSD Foundation
During truncate, fusefs was discarding entire cached blocks, but it wasn't
zeroing out the unused portion of a final partial block. This resulted in
reads returning stale data.
PR: 233783
Reported by: fsx
Sponsored by: The FreeBSD Foundation
Better Makefile syntax.
Note that this commit is to the project branch, but the review concerns the
merge to head.
Sponsored by: The FreeBSD Foundation
This change takes capsicum-test from upstream and applies some local changes to make the
tests work on FreeBSD when executed via Kyua.
The local modifications are as follows:
1. Make `OpenatTest.WithFlag` pass with the new dot-dot lookup behavior in FreeBSD 12.x+.
2. capsicum-test references a set of helper binaries: `mini-me`, `mini-me.noexec`, and
`mini-me.setuid`, as part of the execve/fexecve tests, via execve, fexecve, and open.
It achieves this upstream by assuming `mini-me*` is in the current directory, however,
in order for Kyua to execute `capsicum-test`, it needs to provide a full path to
`mini-me*`. In order to achieve this, I made `capsicum-test` cache the executable's
path from argv[0] in main(..) and use the cached value to compute the path to
`mini-me*` as part of the execve/fexecve testcases.
3. The capsicum-test test suite assumes that it's always being run on CAPABILITIES enabled
kernels. However, there's a chance that the test will be run on a host without a
CAPABILITIES enabled kernel, so we must check for the support before running the tests.
The way to achieve this is to add the relevant `feature_present("security_capabilities")`
check to SetupEnvironment::SetUp() and skip the tests when the support is not available.
While here, add a check for `kern.trap_enotcap` being enabled. As noted by markj@ in
https://github.com/google/capsicum-test/issues/23, this sysctl being enabled can trigger
non-deterministic failures. Therefore, the tests should be skipped if this sysctl is
enabled.
All local changes have been submitted to the capsicum-test project
(https://github.com/google/capsicum-test) and are in various stages of review.
Please see the following pull requests for more details:
1. https://github.com/google/capsicum-test/pull/35
2. https://github.com/google/capsicum-test/pull/41
3. https://github.com/google/capsicum-test/pull/42
Reviewed by: asomers
Discussed with: emaste, markj
Approved by: emaste (mentor)
MFC after: 2 months
Differential Revision: https://reviews.freebsd.org/D19758
By default, FUSE performs authorization in the server. That means that it's
insecure for the client to reuse FUSE file handles between different users,
groups, or processes. Linux handles this problem by creating a different
FUSE file handle for every file descriptor. FreeBSD can't, due to
differences in our VFS design.
This commit adds credential information to each fuse_filehandle. During
open(2), fusefs will now only reuse a file handle if it matches the exact
same access mode, pid, uid, and gid of the calling process.
PR: 236844
Sponsored by: The FreeBSD Foundation
O_EXEC is useful for fexecve(2) and fchdir(2). Treat it as another fufh
type alongside the existing RDONLY, WRONLY, and RDWR. Prior to r345742 this
would've caused a memory and performance penalty.
PR: 236329
Sponsored by: The FreeBSD Foundation
This test shows how bug 236844 can lead to a privilege escalation when used
with the -o allow_other mount option.
PR: 236844
Sponsored by: The FreeBSD Foundation
Previously fusefs would treat any file opened O_WRONLY as though the
FOPEN_DIRECT_IO flag were set, in an attempt to avoid issuing reads as part
of a RMW write operation on a cached part of the file. However, the FUSE
protocol explicitly allows reads of write-only files for precisely that
reason.
Sponsored by: The FreeBSD Foundation
VOP_GETPAGES is disabled when vfs.fusefs.data_cache_mode=0, causing mmap to
return success but accessing the mapped memory will subsequently segfault.
Sponsored by: The FreeBSD Foundation
Surprisingly, open(..., O_WRONLY | O_CREAT, 0444) should work. POSIX
requires it. But it didn't work in early FUSE implementations. Add a
regression test so that our FUSE driver doesn't make the same mistake.
Sponsored by: The FreeBSD Foundation
fuse_vnop_create must close the newly created file if it can't allocate a
vnode. When it does so, it must use the same file flags for FUSE_RELEASE as
it used for FUSE_OPEN or FUSE_CREATE.
Reported by: Coverity
Coverity CID: 1066204
Sponsored by: The FreeBSD Foundation
The test could occasionally hang if the parent's SIGUSR2 signal arrived
before the child had pause()d. Using POSIX semaphores precludes that
possibility.
Sponsored by: The FreeBSD Foundation
If a FUSE filesystem returns ENOSYS for FUSE_CREATE, then fallback to
FUSE_MKNOD/FUSE_OPEN.
Also, fix a memory leak in the error path of fuse_vnop_create. And do a
little cleanup in fuse_vnop_open.
PR: 199934
Reported by: samm@os2.kiev.ua
Sponsored by: The FreeBSD Foundation
The FUSE protocol allows for LOOKUP to return a cacheable negative response,
which means that the file doesn't exist and the kernel can cache its
nonexistence. As of this commit fusefs doesn't cache the nonexistence, but
it does correctly handle such responses. Prior to this commit attempting to
create a file, even with O_CREAT would fail with ENOENT if the daemon
returned a cacheable negative response.
PR: 236231
Sponsored by: The FreeBSD Foundation
For an unknown reason, fusefs was _always_ sending the fdatasync operation
instead of fsync. Now it correctly sends one or the other.
Also, remove the Fsync.fsync_metadata_only test, along with the recently
removed Fsync.nop. They should never have been added. The kernel shouldn't
keep track of which files have dirty data; that's the daemon's job.
PR: 236473
Sponsored by: The FreeBSD Foundation
It's sufficient to check for /dev/fuse. And due to bug 236647, the module
could be named either fuse or fusefs.
PR: 236647
Sponsored by: The FreeBSD Foundation
Also, fix one of the default_permissions test cases. I forgot the
expectation for FUSE_ACCESS, because that doesn't work right now.
Sponsored by: The FreeBSD Foundation
Now the entire fuse test suite can "pass", or at least not fail. Skipped
tests are reported to Kyua as passes, because googletest is still using
Kyua's plain test adapter.
Sponsored by: The FreeBSD Foundation
Sometimes the fuse daemon doesn't die as soon as its /dev/fuse file
descriptor is closed; it needs to be unmounted first.
Sponsored by: The FreeBSD Foundation
Revision r345269 introduced changes that triggered a regression on netmap
unit tests (tests/sys/netmap/ctrl-api-test.c).
This change updates the unit tests to remove the regression.
Reported by: lwhsu
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D19639
This commit adds tests for the default_permissions and push_symlinks_in
mount options. It doesn't add tests for allow_other, because I'm not sure
how that will interact with Kyua (the test will need to drop privileges).
All of the other mount options are undocumented.
PR: 216391
Sponsored by: The FreeBSD Foundation
This mutes the duplicate target warning emitted via bsd.files.mk each build.
MFC after: 1 week
Reviewed by: asomers
Approved by: emaste (mentor)
Differential Revision: https://reviews.freebsd.org/D19603
* Test that FUSE_FLUSH and FUSE_RELEASE release POSIX file locks
* Test that FUSE_SETATTR's attr caching feature works
* Fix some minor mistakes in the posix file lock tests
Sponsored by: The FreeBSD Foundation
This required changing the way that all operations are mocked. Previously
MockFS::process had one input argument and one output argument. Now, it
returns a vector of zero or more responses. This allows tests to simulate
conditions where the filesystem daemon has a queue depth > 1.
PR: 236530
Sponsored by: The FreeBSD Foundation
Make the tests run slightly faster by having pft_ping.py end the capture
of packets as soon as it sees the expected packet, rather than
continuing to sniff.
MFC after: 2 weeks
There was a problem destroying renamed tun interfaces in vnet jails. This was
fixed in r344794. Test the previously failing scenario.
PR: 235704
MFC after: 2 weeks
The netipsec and pf tests have a number of common test functions. These
used to be duplicated, but it makes more sense for them to re-use the
common functions.
PR: 236223
This is marginally faster than using an environment check in each test case.
Also, if the global check fails then all of the tests are skipped. Oddly,
it's not possible to skip a test in any other way.
Also, allow the test to run as a normal user if vfs.usermount=1 and
/dev/fuse is accessible.
Reported by: ngie
Sponsored by: The FreeBSD Foundation
It only tests the kernel portion of fuse, not the userspace portion (which
comes from sysutils/fusefs-libs). The kernel-userspace interface is
de-facto standardized, and this test suite seeks to validate FreeBSD's
implementation.
It uses GoogleMock to substitute for a userspace daemon and validate the
kernel's behavior in response to filesystem access. GoogleMock is
convenient because it can validate the order, number, and arguments of each
operation, and return canned responses.
But that also means that the test suite must use GoogleTest, since
GoogleMock is incompatible with atf-c++ and atf.test.mk does not allow C++
programs to use atf-c.
This commit adds the first 10 test cases out of an estimated 130 total.
PR: 235775, 235773
Sponsored by: The FreeBSD Foundation
Generate a fragmented packet with different header chains, to provoke
the incorrect behaviour of pf.
Without the fix this will trigger a panic.
Obtained from: Corentin Bayet, Nicolas Collignon, Luca Moro at Synacktiv
pfctl has an issue with 'set skip on <group>', which causes inconsistent
behaviour: the set skip directive works initially, but does not take
effect when the same rules are re-applied.
PR: 229241
MFC after: 1 week
When building with KCOV enabled the compiler will insert function calls
to probes allowing us to trace the execution of the kernel from userspace.
These probes are on function entry (trace-pc) and on comparison operations
(trace-cmp).
Userspace can enable the use of these probes on a single kernel thread with
an ioctl interface. It can allocate space for the probe with KIOSETBUFSIZE,
then mmap the allocated buffer and enable tracing with KIOENABLE, with the
trace mode being passed in as the int argument. When complete KIODISABLE
is used to disable tracing.
The first item in the buffer is the number of trace event that have
happened. Userspace can write 0 to this to reset the tracing, and is
expected to do so on first use.
The format of the buffer depends on the trace mode. When in PC tracing just
the return address of the probe is stored. Under comparison tracing the
comparison type, the two arguments, and the return address are traced. The
former method uses on entry per trace event, while the later uses 4. As
such they are incompatible so only a single mode may be enabled.
KCOV is expected to help fuzzing the kernel, and while in development has
already found a number of issues. It is required for the syzkaller system
call fuzzer [1]. Other kernel fuzzers could also make use of it, either
with the current interface, or by extending it with new modes.
A man page is currently being worked on and is expected to be committed
soon, however having the code in the kernel now is useful for other
developers to use.
[1] https://github.com/google/syzkaller
Submitted by: Mitchell Horne <mhorne063@gmail.com> (Earlier version)
Reviewed by: kib
Testing by: tuexen
Sponsored by: DARPA, AFRL
Sponsored by: The FreeBSD Foundation (Mitchell Horne)
Differential Revision: https://reviews.freebsd.org/D14599
Import the unit tests from upstream (https://github.com/luigirizzo/netmap
ba02539859d46d33), and make them ready for use with Kyua.
There are currently 38 regression tests, which test the kernel control ABI
exposed by netmap to userspace applications:
1: test for port info get
2-5: tests for basic port registration
6-9: tests for VALE
10-11: tests for getting netmap allocator info
12-15: tests for netmap pipes
16: test on polling mode
17-18: tests on options
19-27: tests for sync-kloop subsystem
28-39: tests for null ports
31-38: tests for the legacy NIOCREGIF registers
Reviewed by: ngie
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D18490
MK_AUDIT already controls auditd(8), praudit(1), etc. It should also control
the audit test suite.
Submitted by: ngie
MFC after: 2 weeks
Pull Request: https://github.com/freebsd/freebsd/pull/240
These tests should be skipped if /etc/rc.d/auditd is missing, which could be
the case if world was built with WITHOUT_AUDIT set. Also, one test case
requires /etc/rc.d/accounting.
Submitted by: ngie
MFC after: 2 weeks
Pull Request: https://github.com/freebsd/freebsd/pull/240
It's been reported that pf doesn't handle running out of available ports
for NAT correctly. It freezes until a state expires and it can find a
free port.
Test for this, by setting up a situation where only two ports are
available for NAT and then attempting to create three connections.
If successful the third connection will fail immediately. In an
incorrect case the connection attempt will freeze, also freezing all
interaction with pf through pfctl and trigger timeout.
PR: 233867
MFC after: 2 weeks
Use ATF_TC_CLEANUP(), because that means the cleanup code will get
called even if a test fails. Before it would only be executed if every
test within the body succeeded.
Reported by: Marie Helene Kvello-Aune <marieheleneka@gmail.com>
MFC after: 2 weeks
Explicitly mark these tests as requiring root rights. We need to be able
to open /dev/pf.
Reported by: Marie Helene Kvello-Aune <marieheleneka@gmail.com>
MFC after: 2 weeks
Once a signal's siginfo was copied to 'td_si' as part of the signal
exchange in issignal(), it was never cleared. This caused future
thread events that are reported as SIGTRAP events without signal
information to report the stale siginfo in 'td_si'. For example, if a
debugger created a new process and used SIGSTOP to stop it after
PT_ATTACH, future system call entry / exit events would set PL_FLAG_SI
with the SIGSTOP siginfo in pl_siginfo. This broke 'catch syscall' in
current versions of gdb as it assumed PL_FLAG_SI with SIGTRAP
indicates a breakpoint or single step trap.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D18487
Re-apply r341665 with format strings fixed.
If we happen to taste a stale mirror component first, don't reject valid,
newer components that have differing metadata from the stale component
(during STARTING). Instead, update our view of the most recent metadata as
we taste components.
Like mediasize beforehand, remove some checks from g_mirror_check_metadata
which would evict valid components due to metadata that can change over a
mirror's lifetime. g_mirror_check_metadata is invoked long before we check
genid/syncid and decide which component(s) are newest and whether or not we
have quorum.
Before checking if we can enter RUNNING (i.e., we have quorum) after a NEW
component is added, first remove any known stale or inconsistent disks from
the mirrorset, rather than removing them *after* deciding we have quorum.
Check if we have quorum after removing these components.
Additionally, add a knob, kern.geom.mirror.launch_mirror_before_timeout, to
force gmirrors to wait out the full timeout (kern.geom.mirror.timeout)
before transitioning from STARTING to RUNNING. This is a kludge to help
ensure all eligible, boot-time available mirror components are tasted before
RUNNING a gmirror.
Add a basic test case for STARTING -> RUNNING startup behavior around stale
genids.
PR: 232671, 232835
Submitted by: Cindy Yang <cyang AT isilon.com> (previous version)
Reviewed by: markj (kernel portions)
Discussed with: asomers, Cindy Yang
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D18062
r341392 changed common test cleanup routines in a way that allowed them to
be used by TAP tests as well as ATF tests. However, a late change made
during code review resulted in cleanup being broken for ATF tests, which
source geom_subr.sh separately during the body and cleanup phases of the
test. The result was that md(4) devices wouldn't get cleaned up.
MFC after: 2 weeks
X-MFC-With: 341392
If we happen to taste a stale mirror component first, don't reject valid,
newer components that have differing metadata from the stale component
(during STARTING). Instead, update our view of the most recent metadata as
we taste components.
Like mediasize beforehand, remove some checks from g_mirror_check_metadata
which would evict valid components due to metadata that can change over a
mirror's lifetime. g_mirror_check_metadata is invoked long before we check
genid/syncid and decide which component(s) are newest and whether or not we
have quorum.
Before checking if we can enter RUNNING (i.e., we have quorum) after a NEW
component is added, first remove any known stale or inconsistent disks from
the mirrorset, rather than removing them *after* deciding we have quorum.
Check if we have quorum after removing these components.
Additionally, add a knob, kern.geom.mirror.launch_mirror_before_timeout, to
force gmirrors to wait out the full timeout (kern.geom.mirror.timeout)
before transitioning from STARTING to RUNNING. This is a kludge to help
ensure all eligible, boot-time available mirror components are tasted before
RUNNING a gmirror.
When we are instructed to forget mirror components, bump the generation id
to avoid confusion with such stale components later.
Add a basic test case for STARTING -> RUNNING startup behavior around stale
genids.
PR: 232671, 232835
Submitted by: Cindy Yang <cyang AT isilon.com> (previous version)
Reviewed by: markj (kernel portions)
Discussed with: asomers, Cindy Yang
Tested by: pho
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D18062
The problem with the logic prior to this commit was twofold:
1. The wrong set of idioms (TAP-compatible) were being applied to the ATF
testcases when run, resulting in confusing ATF failure results on setup.
2. The cleanup subroutines were broken when the geom classes could not be
loaded as they exited with 0 unexpectedly.
This commit changes the test code to source the class-specific configuration
(conf.sh) once globally, instead of sourcing it per testcase and per cleanup
subroutine, and to call the ATF-specific setup subroutine(s) inline in
the testcases.
The refactoring done is effectively a no-op for the TAP testcases, modulo
any refactoring done to create common code between the ATF and TAP
testcases.
This unbreaks the geli testcases converted to ATF in r327662 and r327683,
and the gmirror testcases added in r327780, respectively, when the geom
class could not be loaded.
tests/sys/geom/class/mirror/...
While here, ignore errors when turning debug failpoint sysctl off, which
could occur if the gmirror class was not loaded.
Submitted by: ngie
MFC after: 2 weeks
Pull Request: https://github.com/freebsd/freebsd/pull/241
Fix reporting of SS_ONSTACK in nested signal delivery when sigaltstack()
is used on some architectures.
Add a unit test for this. I tested the test by introducing the bug
on amd64. I did not test it on other architectures.
Reviewed by: kib
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D18347
After r337820, which "corrected" some spaces-instead-of-tab whitespace
issues in the libkqueue tests, jmg@ pointed out that these files were
originally space-based, not tab-spaced, and so the correction should
have been to get rid of the tabs that had been introduced in previous
changes, not the spaces. This change does that. This is a whitespace
only change; no functional change is intended.
Reported by: jmg@
MFC after: 3 days
Sponsored by: Dell EMC Isilon
Replace hard-coded epair0b with the variable holds the real epair interface
used for testing.
Reviewed by: kp
Approved by: emaste, markj (mentors)
MFC with: r339836
Sponsored by: The FreeBSD Foundation
Set up two jails, configure pfsync between them and create state in one
of them, verify that this state is copied to the other jail.
MFC after: 2 weeks
Sponsored by: Orange Business Services
Differential Revision: https://reviews.freebsd.org/D17504
Unconditionally reparenting to PID 1 breaks the procctl(2) reaper
functionality.
Add a regression test for this case.
Reviewed by: kib
Approved by: re (gjb)
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17589
Originally, these tests accidentally used broadcast addresses when they
should've used unicast addresses. That the tests passed prior to r337736
was accidental.
Submitted by: ae
Reviewed by: olivier
MFC after: 2 weeks
Two of these testcases require software crypto to be enabled. Curiously, it
isn't by default.
PR: 230671
Reported by: Jenkins
Reviewed by: cem
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D16755
FreeBSD's mkstemp sets the temporary file's permissions to 600, and has ever
since mkstemp was added in 1987. Coverity's warning is still relevant for
portable programs since OpenGroup does not require that behavior, and POSIX
didn't until 2008. But none of these programs are portable.
umask(2) should always be used prior to mkstemp(3) so the temporary file
won't be created with insecure permissions.
Reported by: Coverity
CID: 1331605 1347173 1375366 1339800 1331604 1296056 1296060
CID: 1296057 1296062
MFC after: 2 weeks
A follow-up to r337812 to catch a couple more memory leaks that should
have been included in that change.
Reported by: Coverity
CID: 1296064, 1296067 (for real this time)
MFC after: 3 days
X-MFC-with: r337812
Sponsored by: Dell EMC
The libkqueue tests have several places that leak memory by using an
idiom like:
puts(kevent_to_str(kevp));
Rework to save the pointer returned from kevent_to_str() and then
free() it after it has been used.
Reported by: asomers (pointer to Coverity), Coverity
CID: 1296063, 1296064, 1296065, 1296066, 1296067, 1350287, 1394960
Sponsored by: Dell EMC
- Add some more cases to the truncation test.
- Remove the "expect fail" annotations.
PR: 131876
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D16562
With r337328, the test hangs becase the sendmsg() call will block until
the receive buffer is at least partially drained. Fix the problem by
using a non-blocking socket and allowing short writes. Also assert
that a SCM_CREDS message was received if one was expected.
PR: 181741
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D16516
Enable the LOCAL_PEERCRED socket option for unix domain stream sockets
created with socketpair(2). Previously, it only worked with unix domain
stream sockets created with socket(2)/listen(2)/connect(2)/accept(2).
PR: 176419
Reported by: Nicholas Wilson <nicholas@nicholaswilson.me.uk>
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D16350
- Remove return statements in functions with a void return type.
- Allocate enough space for the SCM_CREDS and SCM_RIGHTS messages
received in the rights_creds_payload test.
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
If an error occurs while copying a SCM_RIGHTS message to userspace,
we free the mbuf containing externalized rights, leaking them.
PR: 131876
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
ian@ pointed out that in the test_abstime() function time(NULL) is
used twice; once in an "if" test and again in the enclosed error
message. If the true branch was taken and the process got preempted
before the second time(NULL) call, by the time the error message was
generated enough time could have elapsed that the message could claim
that the event came "too early" but print an event time that was after
the expected timeout. Correct by making the time(NULL) call only once
and using that returned time in both the "if" test and the error
message.
Reported by: ian@
MFC after: 4 days
X-MFC-with: r336761, r336781, r336802
Sponsored by: Dell EMC
Another cast for printing an intmax_t was needed in a kqueue test for
some arches.
Pointy-hat: me (twice)
MFC after: 1 week
X-MFC-with: r336761, r336781
Sponsored by: Dell EMC
If a timer is updated (re-added) with a different time period
(specified in the .data field of the kevent), the new time period has
no effect; the timer will not expire until the original time has
elapsed. This violates the documented behavior as the kqueue(2) man
page says (in part) "Re-adding an existing event will modify the
parameters of the original event, and not result in a duplicate
entry."
This modification, adapted from a patch submitted by cem@ to PR214987,
fixes the kqueue system to allow updating a timer entry. The
kevent timer behavior is changed to:
* When a timer is re-added, update the timer parameters to and
re-start the timer using the new parameters.
* Allow updating both active and already expired timers.
* When the timer has already expired, dequeue any undelivered events
and clear the count of expirations.
All of these changes address the original PR and also bring the
FreeBSD and macOS kevent timer behaviors into agreement.
A few other changes were made along the way:
* Update the kqueue(2) man page to reflect the new timer behavior.
* Fix man page style issues in kqueue(2) diagnosed by igor.
* Update the timer libkqueue system test to test for the updated
timer behavior.
* Fix the (test) libkqueue common.h file so that it includes
config.h which defines various HAVE_* feature defines, before the
#if tests for such variables in common.h. This enables the use of
the actual err(3) family of functions.
* Fix the usages of the err(3) functions in the tests for incorrect
type of variables. Those were formerly undiagnosed due to the
disablement of the err(3) functions (see previous bullet point).
PR: 214987
Reported by: Brian Wellington <bwelling@xbill.org>
Reviewed by: kib
MFC after: 1 week
Relnotes: yes
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D15778
These syscalls were always supposed to have been auditted, but due to
oversights never were.
PR: 228374
Reported by: aniketp
Reviewed by: aniketp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D16388
In r321967 ngie "fixed" these tests by changing their expectations to match
the device numbers produced by the new ino64 code. But it wasn't the tests
that were broken, it was the kernel. bde fixed the kernel in r335053.
Reported by: Jenkins
MFC after: Never (only applies to >= 12)
These three syscalls aren't currently audited correctly, so the tests are
marked as expected failures.
PR: 228374
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D16379
auditon(2) is an ioctl-like syscall with several different variants, each of
which has a distinct audit event. This commit tests the remaining variants
that weren't tested in r336564.
Submitted by: aniketp
MFC after: 2 weeks
X-MFC-With: 336564
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D16381
auditon(2) is an ioctl-like syscall with several different variants, each of
which has a distinct audit event. Write separate audit(4) tests for each
variant.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D16255
Also, fix a bug in common code that could cause other tests to fail: using
ppoll(2) in combination with buffered I/O for /dev/auditpipe. Fix it by
disabling buffering.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D16099
Includes ntp_adjtime, auditctl, acct, auditon, and clock_settime. Includes
quotactl, mount, nmount, swapon, and swapoff in failure mode only. Success
tests for those syscalls will follow. Also includes reboot(2) in failure
mode only. That one can't be tested in success mode.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15898
Tested syscalls include rfork(2), chdir(2), fchdir(2), chroot(2),
getresuid(2), getresgid(2), setpriority(2), setgroups(2), setpgrp(2),
setrlimit(2), setlogin(2), mlock(2), munlock(2), minherit(2), rtprio(2),
profil(2), ktrace(2), ptrace(2), fork(2), umask(2), setuid(2), setgid(2),
seteuid(2), and setegid(2). The last six are only tested in the success
case, either because they're infalliable or a failure is difficult to cause
on-demand.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15966
Includes utimes(2), futimes(2), lutimes(2), futimesat(2), mprotect(2), and
undelete(2). undelete, for now, is tested only in failure mode.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15893
This was correct in the final version on Phabricator, but somehow I screwed
up applying the patch locally.
Reported by: linimon
Pointy-hat-to: asomers
MFC after: 2 weeks
X-MFC-With: 335307
The ad audit class is for administrative commands. This commit adds test
for settimeofday, adjtime, and getfh.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15861
This commit includes extattr_{get_file, get_fd, get_link, list_file,
list_fd, list_link}. It does not include any syscalls that modify, set, or
delete extended attributes, as those are in a different audit class.
Submitted by: aniketpt
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15859
This revision adds auditability tests for stat, lstat, fstat, and fstatat,
all from the fa audit class. More tests from that audit class will follow.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15709
These syscalls are atypical, because each one corresponds to several
different audit events, and they each pass several different audit class
filters.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15657
The only syscalls in this class are close, closefrom, munmap, and revoke.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15650
The only syscalls in this class are rmdir, unlink, unlinkat, rename, and
renameat. Also, set is_exclusive for all audit(4) tests, because they can
start and stop auditd.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15647
truncate and ftruncate are the only syscalls in this class, apart from
certain variations of open and openat, which will be handled in a different
file.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15640
readlink and readlinkat are the only syscalls in this class. open and
openat are as well, but they'll be handled in a different file. Also, tidy
up the copyright headers of recently added files in this area.
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc. (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15636
This change includes the framework for testing the auditability of various
syscalls, and includes changes for the first 12. The tests will start
auditd(8) if needed, though they'll be much faster if it's already running.
The syscalls tested in this commit include mkdir(2), mkdirat(2), mknod(2),
mknodat(2), mkfifo(2), mkfifoat(2), link(2), linkat(2), symlink(2),
symlinkat(2), rename(2), and renameat(2).
Submitted by: aniketp
MFC after: 2 weeks
Sponsored by: Google, Inc (GSoC 2018)
Differential Revision: https://reviews.freebsd.org/D15286
Previously it was possible to connect a socket (which had the
CAP_CONNECT right) by calling "connectat(AT_FDCWD, ...)" even in
capabilties mode. This combination should be treated the same as a call
to connect (i.e. forbidden in capabilities mode). Similarly for bindat.
Disable connectat/bindat with AT_FDCWD in capabilities mode, fix up the
documentation and add tests.
PR: 222632
Submitted by: Jan Kokemüller <jan.kokemueller@gmail.com>
Reviewed by: Domagoj Stolfa
MFC after: 1 week
Relnotes: Yes
Differential Revision: https://reviews.freebsd.org/D15221
A pipe was was left over from a development version of pdeathsig.c and
is not needed.
Process C waits for a signal that'll be generated when process B
exists. Process B waits for process D to send it a byte via pipe_db
before it exits. Process D sends the byte after it has started
ptrace()ing process C. The point of the test is to show that process C
receives the signal because process B exited, even though C has been
reparented to process D. The pipe pipe_cd isn't doing anything useful
(though in an earlier version of the patch it did). Clean that up by
removing the useless pipe.
Submitted by: Thomas Munro
MFC after: 6 days
Differential revision: https://reviews.freebsd.org/D15214
- ptrace__breakpoint_siginfo tests that a SIGTRAP for a software breakpoint
in userland triggers a SIGTRAP with a signal code of TRAP_BRKPT.
- ptrace__step_siginfo tests that a SIGTRAP reported for a step after
stepping via PT_STEP or PT_SETSTEP has a signal code of TRAP_TRACE.
- Use a single list of platforms to define HAVE_BREAKPOINT for platforms
that expose a functional breakpoint() inline to userland. Replace
existing lists of platform tests with HAVE_BREAKPOINT instead.
- Add support for advancing PC past a breakpoint inserted via breakpoint()
to support the existing ptrace__PT_CONTINUE_different_thread test on
non-x86 platforms (x86 advances the PC past the breakpoint instruction,
but other platforms do not). This is implemented by defining a new
SKIP_BREAK macro which accepts a pointer to a 'struct reg' as its sole
argument and modifies the contents to advance the PC. The intention is
to use it in between PT_GETREGS and PT_SETREGS.
Tested on: amd64, i386, mips (after adding a breakpoint() to mips)
MFC after: 1 month
-> PROC_PDEATHSIG_STATUS for consistency with other procctl(2)
operations names.
Requested by: emaste
Sponsored by: The FreeBSD Foundation
MFC after: 13 days
Allow processes to request the delivery of a signal upon death of
their parent process. Supposed consumer of the feature is PostgreSQL.
Submitted by: Thomas Munro
Reviewed by: jilles, mjg
MFC after: 2 weeks
Differential revision: https://reviews.freebsd.org/D15106
fget_cap() tries to do a cheaper snapshot of a file descriptor without
holding the file descriptor lock. This snapshot does not do a deep
copy of the ioctls capability array, but instead uses a different
return value to inform the caller to retry the copy with the lock
held. However, filecaps_copy() was returning 1 to indicate that a
retry was required, and fget_cap() was checking for 0 (actually
'!filecaps_copy()'). As a result, fget_cap() did not do a deep copy
of the ioctls array and just reused the original pointer. This cause
multiple file descriptor entries to think they owned the same pointer
and eventually resulted in duplicate frees.
The only code path that I'm aware of that triggers this is to create a
listen socket that has a restricted list of ioctls and then call
accept() which calls fget_cap() with a valid filecaps structure from
getsock_cap().
To fix, change the return value of filecaps_copy() to return true if
it succeeds in copying the caps and false if it fails because the lock
is required. I find this more intuitive than fixing the caller in
this case. While here, change the return type from 'int' to 'bool'.
Finally, make filecaps_copy() more robust in the failure case by not
copying any of the source filecaps structure over. This avoids the
possibility of leaking a pointer into a structure if a similar future
caller doesn't properly handle the return value from filecaps_copy()
at the expense of one more branch.
I also added a test case that panics before this change and now passes.
Reviewed by: kib
Discussed with: mjg (not a fan of the extra branch)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D15047
This behavior is already documented by the man page, and suggested by POSIX.
Reviewed by: jhb
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D15099
Do not build or install pf tests if WITHOUT_PF is set. This fixes the build
failure with WITHOUT_PF=yes.
Reported by: Vladimir Zakharov <zakharov.vv@gmail.com>
There was a memory leak in the DIOCRADDTABLES ioctl() code which could
be triggered by trying to add tables with the same name.
Try to provoke this memory leak. It was fixed in r331225.
MFC after: 1 week
Validate the DIOCRGETTABLES, DIOCRGETTSTATS, DIOCRCLRTSTATS and
DIOCRSETTFLAGS ioctls with invalid values. These may succeed (because
the kernel uses the minimally required size, not the specified size),
but should not trigger kernel panics.
MFC after: 1 week
The upstream repository is on github BLAKE2/libb2. Files landed in
sys/contrib/libb2 are the unmodified upstream files, except for one
difference: secure_zero_memory's contents have been replaced with
explicit_bzero() only because the previous implementation broke powerpc
link. Preferential use of explicit_bzero() is in progress upstream, so
it is anticipated we will be able to drop this diff in the future.
sys/crypto/blake2 contains the source files needed to port libb2 to our
build system, a wrapped (limited) variant of the algorithm to match the API
of our auth_transform softcrypto abstraction, incorporation into the Open
Crypto Framework (OCF) cryptosoft(4) driver, as well as an x86 SSE/AVX
accelerated OCF driver, blake2(4).
Optimized variants of blake2 are compiled for a number of x86 machines
(anything from SSE2 to AVX + XOP). On those machines, FPU context will need
to be explicitly saved before using blake2(4)-provided algorithms directly.
Use via cryptodev / OCF saves FPU state automatically, and use via the
auth_transform softcrypto abstraction does not use FPU.
The intent of the OCF driver is mostly to enable testing in userspace via
/dev/crypto. ATF tests are added with published KAT test vectors to
validate correctness.
Reviewed by: jhb, markj
Obtained from: github BLAKE2/libb2
Differential Revision: https://reviews.freebsd.org/D14662
It mistakenly believes the 'static' keyword must come first. Fix PPC,
Sparc64, and maybe MIPS world. Fallout from r331279.
Reported by: tinderbox (results come slowly)
The general idea here is to provide userspace programs with well-defined
sources of entropy, in a fashion that doesn't require opening a new file
descriptor (ulimits) or accessing paths (/dev/urandom may be restricted
by chroot or capsicum).
getrandom(2) is the more general API, and comes from the Linux world.
Since our urandom and random devices are identical, the GRND_RANDOM flag
is ignored.
getentropy(3) is added as a compatibility shim for the OpenBSD API.
truss(1) support is included.
Tests for both system calls are provided. Coverage is believed to be at
least as comprehensive as LTP getrandom(2) test coverage. Additionally,
instructions for running the LTP tests directly against FreeBSD are provided
in the "Test Plan" section of the Differential revision linked below. (They
pass, of course.)
PR: 194204
Reported by: David CARLIER <david.carlier AT hardenedbsd.org>
Discussed with: cperciva, delphij, jhb, markj
Relnotes: maybe
Differential Revision: https://reviews.freebsd.org/D14500
The 'physio' fast-path used by AIO requests on md(4) devices, is not
gated on the unsafe_aio knob. Prior to r327755, some AIO requests could
fail the fast-path and fall back to the slow-path (requests for devices
not supporting unmapped I/O and requests which failed with EFAULT during
the fast-path). However, those cases now return a suitable error rather
than using the slow-path.
PR: 217261
Reviewed by: asomers
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D14742
The code assumed that disks (devices) used for testing are always named
like /dev/foo, but there is no reason for that restriction and we can
easily support paths like /dev/stripe/bar.
The change is to account for a different order in which the recursive
destroy may be attempted. If we first try a dataset that can be destroyed
then it will be destroyed, but if we first try a dataset that cannot be
destroyed then we will not attempt to destroy the other dataset.
It was originally written by Sun as part of the STF (Solaris test framework).
They open sourced it in OpenSolaris, then HighCloud partially ported it to
FreeBSD, and Spectra Logic finished the port. We also added many testcases,
fixed many broken ones, and converted them all to the ATF framework. We've had
help along the way from avg, araujo, smh, and brd.
By default most of the tests are disabled. Set the disks Kyua variable to
enable them.
Submitted by: asomers, will, justing, ken, brd, avg, araujo, smh
Sponsored by: Spectra Logic Corp, HighCloud
If the underlying provider's physical path is null, then the gpart device's
physical path will be, too. Otherwise, it will append the partition name,
such as "/p1" or "/s1/a". This will make gpart work better with zfsd(8).
PR: 224965
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D14010
If the underlying provider's physical path is null, then the geli device's
physical path will be, too. Otherwise, it will append "/eli". This will make
geli work better with zfsd(8).
PR: 224962
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D13979
Don't declare some types that FreeBSD incorrectly declares.
Fix an incorrect call to open() (missing mode).
ANSIfy prototypes.
Enable SysV message queue, semaphore, and shared memory tests.
With exception of the workaround for union semun, these fixes have been
committed to NetBSD.
Reviewed by: asomers
Approved by: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D13471
* Registers TRNG source for random(4)
* Finds available queues, LSBs; allocates static objects
* Allocates a shared MSI-X for all queues. The hardware does not have
separate interrupts per queue. Working interrupt mode driver.
* Computes SHA hashes, HMAC. Passes cryptotest.py, cryptocheck tests.
* Does AES-CBC, CTR mode, and XTS. cryptotest.py and cryptocheck pass.
* Support for "authenc" (AES + HMAC). (SHA1 seems to result in
"unaligned" cleartext inputs from cryptocheck -- which the engine
cannot handle. SHA2 seems to work fine.)
* GCM passes for block-multiple AAD, input lengths
Largely based on ccr(4), part of cxgbe(4).
Rough performance averages on AMD Ryzen 1950X (4kB buffer):
aesni: SHA1: ~8300 Mb/s SHA256: ~8000 Mb/s
ccp: ~630 Mb/s SHA256: ~660 Mb/s SHA512: ~700 Mb/s
cryptosoft: ~1800 Mb/s SHA256: ~1800 Mb/s SHA512: ~2700 Mb/s
As you can see, performance is poor in comparison to aesni(4) and even
cryptosoft (due to high setup cost). At a larger buffer size (128kB),
throughput is a little better (but still worse than aesni(4)):
aesni: SHA1:~10400 Mb/s SHA256: ~9950 Mb/s
ccp: ~2200 Mb/s SHA256: ~2600 Mb/s SHA512: ~3800 Mb/s
cryptosoft: ~1750 Mb/s SHA256: ~1800 Mb/s SHA512: ~2700 Mb/s
AES performance has a similar story:
aesni: 4kB: ~11250 Mb/s 128kB: ~11250 Mb/s
ccp: ~350 Mb/s 128kB: ~4600 Mb/s
cryptosoft: ~1750 Mb/s 128kB: ~1700 Mb/s
This driver is EXPERIMENTAL. You should verify cryptographic results on
typical and corner case inputs from your application against a known- good
implementation.
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12723
Scapy requires the Raw payload to be a string, which was not the case. This
caused the pft_ping.py script to fail, which in turn caused the test to fail.
Reduce the geli tests' runtime by about a third:
* In integrity_test:copy, use a file-backed md(4) device instead of a
malloc'd one. That way we can corrupt the underlying storage without
needing to detach and reattach the geli device.
* In integrity_test:{copy, hmac, data} and onetime_test:{onetime,
onetime_a}, move reads of /dev/random out of the loop.
MFC after: 2 weeks
