Historically we have not distinguished between kernel wirings and user
wirings for accounting purposes. User wirings (via mlock(2)) were
subject to a global limit on the number of wired pages, so if large
swaths of physical memory were wired by the kernel, as happens with
the ZFS ARC among other things, the limit could be exceeded, causing
user wirings to fail.
The change adds a new counter, v_user_wire_count, which counts the
number of virtual pages wired by user processes via mlock(2) and
mlockall(2). Only user-wired pages are subject to the system-wide
limit which helps provide some safety against deadlocks. In
particular, while sources of kernel wirings typically support some
backpressure mechanism, there is no way to reclaim user-wired pages
shorting of killing the wiring process. The limit is exported as
vm.max_user_wired, renamed from vm.max_wired, and changed from u_int
to u_long.
The choice to count virtual user-wired pages rather than physical
pages was done for simplicity. There are mechanisms that can cause
user-wired mappings to be destroyed while maintaining a wiring of
the backing physical page; these make it difficult to accurately
track user wirings at the physical page layer.
The change also closes some holes which allowed user wirings to succeed
even when they would cause the system limit to be exceeded. For
instance, mmap() may now fail with ENOMEM in a process that has called
mlockall(MCL_FUTURE) if the new mapping would cause the user wiring
limit to be exceeded.
Note that bhyve -S is subject to the user wiring limit, which defaults
to 1/3 of physical RAM. Users that wish to exceed the limit must tune
vm.max_user_wired.
Reviewed by: kib, ngie (mlock() test changes)
Tested by: pho (earlier version)
MFC after: 45 days
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D19908
Initialize `oldlen` to the size of the value, instead of leaving the value
unitialized. Leaving it unitialized seems to work by accident on amd64 when
running 64-bit programs, but not on i386.
This matches patterns in use in other programs.
PR: 237458
Approved by: emaste (mentor; implicit)
MFC after: 1 week
Tested on: ^/head (amd64), ^/stable/11 (i386)
The current logic for CSTD/CXXSTD requires homogenity as far as the
supported C/C++ standards, which is a sensible default. However, when
dealing with differing versions of C++, some code may compile with C++11, but
not C++17 (for instance). So in order to avoid having people convert over their
code to the new standard, give the users the ability to specify the standard on
a per-program basis.
This will allow a user to override the supporting standard for a set of
programs, mixing C++11 with C++14 (for instance).
Reviewed by: asomers
Apprved by: emaste (mentor)
MFC after: 1 month
MFC with: r345708
Differential Revision: https://reviews.freebsd.org/D19738
When a review is closed via Phabricator it updates the patch attached to the
review. I downloaded the raw patch from Phabricator, applied it, and repeated
my mistake from r345704 by accident mixing content from D19732 and D19738.
For my own personal sanity, I will try not to mix reviews like this in the
future.
MFC after: 1 month
MFC with: r345706
Approved by: emaste (mentor, implicit)
CXXSTD was added as the C++ analogue to CSTD.
CXXSTD defaults to `-std=c++11` with supporting compilers; `-std=gnu++98`,
otherwise for older versions of g++.
This change standardizes the CXXSTD variable, originally added to
googletest.test.inc.mk as part of r345203.
As part of this effort, convert all `CXXFLAGS+= -std=*` calls to use `CXXSTD`.
Notes:
This value is not sanity checked in bsd.sys.mk, however, given the two
most used C++ compilers on FreeBSD (clang++ and g++) support both modes, it is
likely to work with both toolchains. This method will be refined in the future
to support more variants of C++, as not all versions of clang++ and g++ (for
instance) support C++14, C++17, etc.
Any manual appending of `-std=*` to `CXXFLAGS` should be replaced with CXXSTD.
Example:
Before this commit:
```
CXXFLAGS+= -std=c++14
```
After this commit:
```
CXXSTD= c++14
```
Reviewed by: asomers
Approved by: emaste (mentor)
MFC after: 1 month
MFC with: r345203, r345704, r345705
Relnotes: yes
Tested with: make tinderbox
Differential Revision: https://reviews.freebsd.org/D19732
I accidentally committed code from two reviews. I will reintroduce the code to
bsd.progs.mk as part of a separate commit from r345704.
Approved by: emaste (mentor, implicit)
MFC after: 2 months
MFC with: r345704
CXXSTD defaults to `-std=c++11` with supporting compilers; `-std=gnu++98`,
otherwise for older versions of g++.
This change standardizes the CXXSTD variable, originally added to
googletest.test.inc.mk as part of r345203.
As part of this effort, convert all `CXXFLAGS+= -std=*` calls to use `CXXSTD`.
Notes:
This value is not sanity checked in bsd.sys.mk, however, given the two
most used C++ compilers on FreeBSD (clang++ and g++) support both modes, it is
likely to work with both toolchains. This method will be refined in the future
to support more variants of C++, as not all versions of clang++ and g++ (for
instance) support C++14, C++17, etc.
Any manual appending of `-std=*` to `CXXFLAGS` should be replaced with CXXSTD.
Example:
Before this commit:
```
CXXFLAGS+= -std=c++14
```
After this commit:
```
CXXSTD= c++14
```
Reviewed by: asomers
Approved by: emaste (mentor)
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D19732
In short, the prior code was far too simplistic when it came to calling recv(2)
and failed intermittently (or in the case of Jenkins, deterministically).
Handle short recv(2)s by checking the return code and incrementing the window
into the buffer by the number of received bytes. If the number of received
bytes <= 0, then bail out of the loop, and test the total number of received
bytes vs the expected number of bytes sent for equality, and base whether or
not the test passes/fails on that fact.
Remove the expected failure, now that the hdtr testcases deterministically pass
on my host after this change [1].
PR: 234809 [1], 235200
Reviewed by: asomers
Approved by: emaste (mentor)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D19188
ci.FreeBSD.org does not have access to a DNS resolver/network (unlike my test
VM), so in order for the test to pass on the host, it needs to avoid the DNS
lookup by using the numeric host address representation.
PR: 235200
Reviewed by: asomers, lwhsu
Approved by: emaste (mentor)
MFC after: 2 weeks
MFC with: r343362, r343365, r343367-r343368, r343461
Differential Revision: https://reviews.freebsd.org/D19026
The return value (`err`) should be checked; not the `errno` value.
PR: 235200
Approved by: emaste (mentor)
Reviewed by: asomers, lwhsu
MFC after: 28 days
MFC with: r343362, r343365, r343367-r343368
Differential Revision: https://reviews.freebsd.org/D18969
I should have only changed the format qualifier with the `size_t` value,
`length`, not the other [`off_t`] value, `dest_file_size`.
MFC after: 1 month
MFC with: r343362, r343365, r343367
Approved by: emaste (mentor; implicit)
Reported by: gcc 8.x
gcc 8.x is more pedantic than clang 7.x with format strings and the tests
passed `void*` variables while supplying `%s` (which is technically
incorrect).
Make the affected `void*` variables use `char*` storage instead to address
this issue, as the compiler will upcast the values to `char*`.
MFC after: 1 month
MFC with: r343362
Approved by: emaste (mentor; implicit)
Reviewed by: asomers
Differential Revision: https://reviews.freebsd.org/D18934
These testcases exercise a number of functional requirements for sendfile(2).
The testcases use IPv4 and IPv6 domain sockets with TCP, and were confirmed
functional on UFS and ZFS. UDP address family sockets cannot be used per the
sendfile(2) contract, thus using UDP sockets is outside the scope of
testing the syscall in positive cases. As seen in
`:s_negative_udp_socket_test`, UDP is used to test the sendfile(2) contract
to ensure that EINVAL is returned by sendfile(2).
The testcases added explicitly avoid testing out `SF_SYNC` due to the
complexity of verifying that support. However, this is a good next logical
item to verify.
The `hdtr_positive*` testcases work to a certain degree (the header
testcases pass), but the trailer testcases do not work (it is an expected
failure). In particular, the value received by the mock server doesn't match
the expected value, and instead looks something like the following (using
python array notation):
`trailer[:]message[1:]`
instead of:
`message[:]trailer[:]`
This makes me think there's a buffer overrun issue or problem with the
offset somewhere in the sendfile(2) system call, but I need to do some
other testing first to verify that the code is indeed sane, and my
assumptions/code isn't buggy.
The `sbytes_negative` testcases that check `sbytes` being set to an
invalid value resulting in `EFAULT` fails today as the other change
(which checks `copyout(9)`) has not been committed [1]. Thus, it
should remain an expected failure (see bug 232210 for more details
on this item).
Next steps for testing sendfile(2):
1. Fix the header/trailer testcases so that they pass.
2. Setup if_tap interface and test with it, instead of using "localhost", per
@asomers's suggestion.
3. Handle short recv(2)'s in `server_cat(..)`.
4. Add `SF_SYNC` support.
5. Add some more negative tests outside the scope of the functional contract.
MFC after: 1 month
Reviewed by: asomers
Approved by: emaste (mentor)
PR: 232210
Sponsored by: Netflix, Inc
Differential Revision: https://reviews.freebsd.org/D18625
This fixes the obscure endless loop seen with case-insensitive
patterns containing characters in 128-255 range; originally
found running GNU grep test suite.
Our regex implementation being kludgy translates the characters
in case-insensitive pattern to bracket expression containing both
cases for the character and doesn't correctly handle the case when
original character is in bitmap and the other case is not, falling
into the endless loop going through in p_bracket(), ordinary(),
and bothcases().
Reducing the bitmap to 0-127 range for multibyte locales solves this
as none of these characters have other case mapping outside of bitmap.
We are also safe in the case when the original character outside of
bitmap has other case mapping in the bitmap (there are several of those
in our current ctype maps having unidirectional mapping into bitmap).
Reviewed by: bapt, kevans, pfg
Differential revision: https://reviews.freebsd.org/D18302
Matcher function incorrectly assumed that moffset that we get from
findmust is in bytes. Fix this by introducing a stepback function,
taking short path if MB_CUR_MAX is 1, and going back byte-by-byte,
checking if we have a legal character sequence otherwise.
PR: 153502
Reviewed by: pfg, kevans
Approved by: kib (mentor, implicit)
Differential revision: https://reviews.freebsd.org/D18297
Currently libclang_rt is not provided for cross-building and as such
is not connected to cross-tools. For building clang once in universe
it is likely that libclang_rt won't exist for the universe toolchain
but even if it did it would not support anything but the native arch.
So explicitly check for support before enabling h_raw.
MFC after: 1 week
Reviewed by: dim
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D16012
Previously, libc.so would initialize its notion of the break address
using _end, a special symbol emitted by the static linker following
the bss section. Compatibility issues between lld and ld.bfd could
cause the wrong definition of _end (libc.so's definition rather than
that of the executable) to be used, breaking the brk()/sbrk()
interface.
Avoid this problem and future interoperability issues by simply not
relying on _end. Instead, modify the break() system call to return
the kernel's view of the current break address, and have libc
initialize its state using an extra syscall upon the first use of the
interface. As a side effect, this appears to fix brk()/sbrk() usage
in executables run with rtld direct exec, since the kernel and libc.so
no longer maintain separate views of the process' break address.
PR: 228574
Reviewed by: kib (previous version)
MFC after: 2 months
Differential Revision: https://reviews.freebsd.org/D15663
On older kernels, when userspace program disables SIGSYS, catch ENOSYS and
emulate getrandom(2) syscall with the kern.arandom sysctl (via existing
arc4_sysctl wrapper).
Special care is taken to faithfully emulate EFAULT on NULL pointers, because
sysctl(3) as used by kern.arandom ignores NULL oldp. (This was caught by
getentropy(3) ATF tests.)
Reported by: kib
Reviewed by: kib
Discussed with: delphij
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D14785
The general idea here is to provide userspace programs with well-defined
sources of entropy, in a fashion that doesn't require opening a new file
descriptor (ulimits) or accessing paths (/dev/urandom may be restricted
by chroot or capsicum).
getrandom(2) is the more general API, and comes from the Linux world.
Since our urandom and random devices are identical, the GRND_RANDOM flag
is ignored.
getentropy(3) is added as a compatibility shim for the OpenBSD API.
truss(1) support is included.
Tests for both system calls are provided. Coverage is believed to be at
least as comprehensive as LTP getrandom(2) test coverage. Additionally,
instructions for running the LTP tests directly against FreeBSD are provided
in the "Test Plan" section of the Differential revision linked below. (They
pass, of course.)
PR: 194204
Reported by: David CARLIER <david.carlier AT hardenedbsd.org>
Discussed with: cperciva, delphij, jhb, markj
Relnotes: maybe
Differential Revision: https://reviews.freebsd.org/D14500
In contrast to the existing NetBSD setcontext_link test, these tests
verify that passing from 1 to 6 arguments through to the callback function
work correctly which can be useful for testing ABIs which split arguments
between registers and the stack.
Sponsored by: DARPA / AFRL
libregex is a regex(3) implementation intended to feature GNU extensions and
any other non-POSIX compliant extensions that are deemed worthy.
These extensions are separated out into a separate library for the sake of
not cluttering up libc further with them as well as not deteriorating the
speed (or lack thereof) of the libc implementation.
libregex is implemented as a build of the libc implementation with LIBREGEX
defined to distinguish this from a libc build. The reasons for
implementation like this are two-fold:
1.) Maintenance- This reduces the overhead induced by adding yet another
regex implementation to base.
2.) Ease of use- Flipping on GNU extensions will be as simple as linking
against libregex, and POSIX-compliant compilations can be guaranteed with a
REG_POSIX cflag that should be ignored by libc/regex and disables extensions
in libregex. It is also easier to keep REG_POSIX sane and POSIX pure when
implemented in this fashion.
Tests are added for future functionality, but left disconnected for the time
being while other testing is done.
Reviewed by: cem (previous version)
Differential Revision: https://reviews.freebsd.org/D12934
Currently each call to telldir() requires a malloc and adds an entry to a
linked list which must be traversed on future telldir(), seekdir(),
closedir(), and readdir() calls. Applications that call telldir() for every
directory entry incur O(n^2) behavior in readdir() and O(n) in telldir() and
closedir().
This optimization eliminates the malloc() and linked list in most cases by
packing the relevant information into a single long. On 64-bit architectures
msdosfs, NFS, tmpfs, UFS, and ZFS can all use the packed representation. On
32-bit architectures msdosfs, NFS, and UFS can use the packed
representation, but ZFS and tmpfs can only use it for about the first 128
files per directory. Memory savings is about 50 bytes per telldir(3) call.
Speedup for telldir()-heavy directory traversals is about 20-30x for one
million files per directory.
Reviewed by: kib, mav, mckusick
MFC after: 3 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D13385
getmntinfo(3) is designed around a relatively static or slow growing set of
current mounts. It tried to detect a race with somewhat concurrent mount
and re-call getfsstat(2) in that case, looping indefinitely. It also
allocated space for a single extra mount as slop.
In the case where the user has a large number of mounts and is adding them
at a rapid pace, it fell over.
This patch makes two functional changes:
1. Allocate even more slop. Double whatever the last getfsstat(2) returned.
2. Abort and return some known results after looping a few times
(arbitrarily, 3). If the list is constantly changing, we can't guarantee
we return a full result to the user at any point anyways.
While here, add very basic functional tests for getmntinfo(3) to the libc
suite.
PR: 221743
Submitted by: Peter Eriksson <peter AT ifm.liu.se> (earlier version)
Sponsored by: Dell EMC Isilon
abort_handler_s() currently simply calls abort(), though the standard
specifies more: "Writes an implementation-defined message to stderr
which must include the string pointed to by msg and calls abort()."
memset_s() is missing error condition "n > smax", and does not invoke
the constraint handler after filling the buffer: "following errors are
detected at runtime and call the currently installed constraint
handler function after storing ch in every location of the destination
range [dest, dest+destsz) if dest and destsz are themselves valid",
one of the errors is "n > smax" itself.
Submitted by: Yuri Pankov <yuripv@gmx.com>
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D11991
ATF cleanup routines run in separate processes from the tests themselves, so
they can't share global variables.
Also, setdomainname_test needs to be is_exclusive because the test cases
access a global resource.
PR: 219967
Reviewed by: ngie
MFC after: 3 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D11188
const char *.
This fixes a bogus set of errors from gcc about strdup not being allowed a NULL
argument.
MFC after: 1 week
Sponsored by: Dell EMC Isilon
- Fix -Wmissing-declaration warning by staticizing run_tests.
- Fix -Wsign-compare warnings by casting size_t types to int
for comparisons.
Reindent some of the code in sdump_hostent(..) to accomodate the
overall changes.
MFC after: 3 days
Sponsored by: Dell EMC Isilon
