reports snap counts of how much a zone alloced and how much it freed. It
may happen that snap values doesn't match, e.g alloced - freed < 0.
Workaround that in memstat library.
Reported by: pho
back to the lever before r343030. For 64-bit machines reduce it slightly,
too. Together with r343030 I bumped the limit up to the value we use at
Netflix to serve 100 Gbit/s of sendfile traffic, and it probably isn't a
good default.
Provide a loader tunable to change vnode pager pbufs count. Document it.
Building binaries as PIE allows the executable itself to be loaded at a
random address when ASLR is enabled (not just its shared libraries).
With this change PIE objects have a .pieo extension and INTERNALLIB
libraries libXXX_pie.a.
MK_PIE is disabled for some kerberos5 tools, Clang, and Subversion, as
they explicitly reference .a libraries in their Makefiles. These can
be addressed on an individual basis later. MK_PIE is also disabled for
rtld-elf because it is already position-independent using bespoke
Makefile rules.
Currently only dynamically linked binaries will be built as PIE.
Discussed with: dim
Reviewed by: kib
MFC after: 1 month
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18423
In particular, use ifuncs for __getcontextx_size(), also calculate the
size of the extended save area in resolver. Same for __fillcontextx2().
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
be_destroy is documented to recursively destroy a boot environment. In the
case of snapshots, one would take this to mean that these are also
recursively destroyed. However, this was previously not the case.
be_destroy would descend into the be_destroy callback and attempt to
zfs_iter_children on the top-level snapshot, which is bogus.
Our alternative approach is to take note of the snapshot name and iterate
through all of fs children of the BE to try destruction in the children.
The -o option is also fixed to work properly with deep BEs. If the BE was
created with `bectl create -e otherDeepBE newDeepBE`, for instance, then a
recursive snapshot of otherDeepBE would have been taken for construction of
newDeepBE but a subsequent destroy with BE_DESTROY_ORIGIN set would only
clean up the snapshot at the root of otherDeepBE: ${BEROOT}/otherDeepBE@...
The most recent iteration instead pretends not to know how these things
work, verifies that the origin is another BE and then passes that back
through be_destroy to DTRT when snapshots and deep BEs may be in play.
MFC after: 1 week
Sync libarchive with vendor.
Relevant vendor changes:
PR #1085: Fix a null pointer dereference bug in zip writer
PR #1110: ZIP reader added support for XZ, LZMA, PPMD8 and BZIP2
decopmpression
PR #1116: Add support for 64-bit ar format
PR #1120: Fix a 7zip crash [1] and a ISO9660 infinite loop [2]
PR #1125: RAR5 reader - fix an invalid read and a memory leak
PR #1131: POSIX reader - do not fail when tree_current_lstat() fails
due to ENOENT [3]
PR #1134: Delete unnecessary null pointer checks before calls of free()
OSS-Fuzz 10843: Force intermediate to uint64_t to make UBSAN happy.
OSS-Fuzz 11011: Avoid buffer overflow in rar5 reader
PR: 233006 [3]
Security: CVE-2019-1000019 [1], CVE-2019-1000020 [2]
MFC after: 2 weeks
nvpair_create_stringv: free the temporary string; this fix affects
nvlist_add_stringf() and nvlist_add_stringv().
nvpair_remove_nvlist_array (NV_TYPE_NVLIST_ARRAY case): free the chain
of nvpairs (as resetting it prevents nvlist_destroy() from freeing it).
Note: freeing the chain in nvlist_destroy() is not sufficient, because
it would still leak through nvlist_take_nvlist_array(). This affects
all nvlist_*_nvlist_array() use
Submitted by: Mindaugas Rasiukevicius <rmind@netbsd.org>
Reported by: clang/gcc ASAN
MFC after: 2 weeks
Currently origin snapshots are left behind when a BE is destroyed, whether
it was an auto-created snapshot or explicitly specified via, for example,
`bectl create -e be@mysnap ...`.
Removing it automatically could be argued as a POLA violation in some
circumstances, so provide a flag to be_destroy for it. An accompanying
option will be added to bectl(8) to utilize this.
Some minor style/consistency nits in the affected areas also addressed.
Reported by: Shawn Webb
MFC after: 1 week
Replace calls to sinf(x) and cosf(x) with a single call to sincosf().
Submitted by: Steve Kargl <sgk@troutmask.apl.washington.edu>
Reviewed by: bde
Approved by: grog
MFC after: 3 days
trig_test.reduction test cases to fail, if the fixes from r343916 have
not yet been applied to the base compiler.
Reported by: lwhsu
PR: 234040
Upstream PR: https://bugs.llvm.org/show_bug.cgi?id=40206
MFC after: 1 week
[X86] Add FPSW as a Def on some FP instructions that were missing it.
Pull in r353141 from upstream llvm trunk (by Craig Topper):
[X86] Connect the default fpsr and dirflag clobbers in inline
assembly to the registers we have defined for them.
Summary:
We don't currently map these constraints to physical register numbers
so they don't make it to the MachineIR representation of inline
assembly.
This could have problems for proper dependency tracking in the
machine schedulers though I don't have a test case that shows that.
Reviewers: rnk
Reviewed By: rnk
Subscribers: eraman, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D57641
Pull in r353489 from upstream llvm trunk (by Craig Topper):
[X86] Add FPCW as a register and start using it as an implicit use on
floating point instructions.
Summary:
FPCW contains the rounding mode control which we manipulate to
implement fp to integer conversion by changing the roudning mode,
storing the value to the stack, and then changing the rounding mode
back. Because we didn't model FPCW and its dependency chain, other
instructions could be scheduled into the middle of the sequence.
This patch introduces the register and adds it as an implciit def of
FLDCW and implicit use of the FP binary arithmetic instructions and
store instructions. There are more instructions that need to be
updated, but this is a good start. I believe this fixes at least the
reduced test case from PR40529.
Reviewers: RKSimon, spatel, rnk, efriedma, andrew.w.kaylor
Subscribers: dim, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D57735
These should fix a problem in clang 7.0 where it would sometimes emit
long double floating point instructions in a slightly wrong order,
leading to failures in our libm tests. In particular, the cbrt_test
test case 'cbrtl_powl' and the trig_test test case 'reduction'.
Also bump __FreeBSD_cc_version, to be able to detect this in our test
suite.
Reported by: lwhsu
PR: 234040
Upstream PR: https://bugs.llvm.org/show_bug.cgi?id=40206
MFC after: 1 week
Back in 1993, the fgetln (then fgetline) interface was changed to not
return a C string. The change was accomplished by ifdefing out the code
that did the termination. Changing the interface would violate our API
stability rules so remove the old implementation.
Sponsored by: DARPA, AFRL
Refactor the function calls and tests so that, on UFS, the proper fields
are filled out.
PR: 233849
Reported by: Andre Albsmeier
Reviewed by: mav, delphij
MFC after: 1 month
Sponsored by: iXsystems Inc
Differential Revision: https://reviews.freebsd.org/D18785
This optimizes out runtime switch and removes yet another cpuid from
libc.
Note that this is the first use of ifunc in i386 libc, so
ifunc-capable toolchain is required for building runnable userspace on
i386, same as on amd64.
Discussed with: emaste
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
from its parent so that LOG_PERROR would work. However, this caused
dhclient(8)'s stdio streams to remain open across daemonization, breaking
the ability to capture its foreground output as done in netconfig_ipv4.
Fix this by reverting r341692 and instead passing the parent's stderr
descriptor as an argument to cap_openlog() only when LOG_PERROR is specified
in logopt.
PR: 234514
Suggested by: markj
Reported by: Shawn Webb
Reviewed by: markj, oshogbo
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D18989
Use recent best practices for Copyright form at the top of
the license:
1. Remove all the All Rights Reserved clauses on our stuff. Where we
piggybacked others, use a separate line to make things clear.
2. Use "Netflix, Inc." everywhere.
3. Use a single line for the copyright for grep friendliness.
4. Use date ranges in all places for our stuff.
Approved by: Netflix Legal (who gave me the form), adrian@ (pmc files)
When libthr is statically linked into the binary, order of the
constructors execution is not deterministic. It is possible for the
application constructor to use pthread_mutex_* functions before the
libthr initialization was done.
Handle it by:
- making thr_malloc.c locking functions operational when curthread is not
yet set;
- making __thr_malloc_init() idempotent, allowing more than one call to it;
- unconditionally calling __thr_malloc_init() before initializing
a process-private mutex.
Reported and tested by: mmel
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
ci.FreeBSD.org does not have access to a DNS resolver/network (unlike my test
VM), so in order for the test to pass on the host, it needs to avoid the DNS
lookup by using the numeric host address representation.
PR: 235200
Reviewed by: asomers, lwhsu
Approved by: emaste (mentor)
MFC after: 2 weeks
MFC with: r343362, r343365, r343367-r343368, r343461
Differential Revision: https://reviews.freebsd.org/D19026
Effectively all i386 kernels now have two pmaps compiled in: one
managing PAE pagetables, and another non-PAE. The implementation is
selected at cold time depending on the CPU features. The vm_paddr_t is
always 64bit now. As result, nx bit can be used on all capable CPUs.
Option PAE only affects the bus_addr_t: it is still 32bit for non-PAE
configs, for drivers compatibility. Kernel layout, esp. max kernel
address, low memory PDEs and max user address (same as trampoline
start) are now same for PAE and for non-PAE regardless of the type of
page tables used.
Non-PAE kernel (when using PAE pagetables) can handle physical memory
up to 24G now, larger memory requires re-tuning the KVA consumers and
instead the code caps the maximum at 24G. Unfortunately, a lot of
drivers do not use busdma(9) properly so by default even 4G barrier is
not easy. There are two tunables added: hw.above4g_allow and
hw.above24g_allow, the first one is kept enabled for now to evaluate
the status on HEAD, second is only for dev use.
i386 now creates three freelists if there is any memory above 4G, to
allow proper bounce pages allocation. Also, VM_KMEM_SIZE_SCALE changed
from 3 to 1.
The PAE_TABLES kernel config option is retired.
In collaboarion with: pho
Discussed with: emaste
Reviewed by: markj
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D18894
The need to use libc malloc(3) from some places in libthr always
caused issues. For instance, per-thread key allocation was switched to
use plain mmap(2) to get storage, because some third party mallocs
used keys for implementation of calloc(3).
Even more important, libthr calls calloc(3) during initialization of
pthread mutexes, and jemalloc uses pthread mutexes. Jemalloc provides
some way to both postpone the initialization, and to make
initialization to use specialized allocator, but this is very fragile
and often breaks. See the referenced PR for another example.
Add the small malloc implementation used by rtld, to libthr. Use it in
thr_spec.c and for mutexes initialization. This avoids the issues with
mutual dependencies between malloc and libthr in principle. The
drawback is that some more allocations are not interceptable for
alternate malloc implementations. There should be not too much memory
use from this allocator, and the alternative, direct use of mmap(2) is
obviously worse.
PR: 235211
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D18988
This includes the bump for cdevsw d_version. Otherwise, the impact on
the ABI (not KBI) is surprisingly low. The most important affected
interface is devname(3) and ttyname(3) which already correctly handle
long names (and ttyname(3) should not be affected at all).
Still, due to the d_version bump, I argue that the change is not MFC-able.
Requested by: mmacy
Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D18932
The strdup() call does not take advantage of the known length of the
source string. Replace by malloc() and memcpy() utilizimng the pre-
calculated string length.
Submitted by: cperciva
Reported by: rgrimes
MFC after: 2 weeks
Casper library should not use exit(3) function because before setting it up
applications may register it. Casper doesn't depend on any registered exit
function, so it safe to change this.
Reported by: jceel
MFC after: 2 weeks
The return value (`err`) should be checked; not the `errno` value.
PR: 235200
Approved by: emaste (mentor)
Reviewed by: asomers, lwhsu
MFC after: 28 days
MFC with: r343362, r343365, r343367-r343368
Differential Revision: https://reviews.freebsd.org/D18969
This is meant to clarify the fact that the system call will not fail
with -1/EFAULT, as one might expect, when reading the sendfile(2)
manpage today.
While here, pet the mandoc linter, when dealing with the section that
describes valid values for `flags`.
PR: 232210
MFC after: 2 weeks
Approved by: emaste (mentor)
Reviewed by: glebius, 0mp
Differential Revision: https://reviews.freebsd.org/D18949
I should have only changed the format qualifier with the `size_t` value,
`length`, not the other [`off_t`] value, `dest_file_size`.
MFC after: 1 month
MFC with: r343362, r343365, r343367
Approved by: emaste (mentor; implicit)
Reported by: gcc 8.x
gcc 8.x is more pedantic than clang 7.x with format strings and the tests
passed `void*` variables while supplying `%s` (which is technically
incorrect).
Make the affected `void*` variables use `char*` storage instead to address
this issue, as the compiler will upcast the values to `char*`.
MFC after: 1 month
MFC with: r343362
Approved by: emaste (mentor; implicit)
Reviewed by: asomers
Differential Revision: https://reviews.freebsd.org/D18934
These testcases exercise a number of functional requirements for sendfile(2).
The testcases use IPv4 and IPv6 domain sockets with TCP, and were confirmed
functional on UFS and ZFS. UDP address family sockets cannot be used per the
sendfile(2) contract, thus using UDP sockets is outside the scope of
testing the syscall in positive cases. As seen in
`:s_negative_udp_socket_test`, UDP is used to test the sendfile(2) contract
to ensure that EINVAL is returned by sendfile(2).
The testcases added explicitly avoid testing out `SF_SYNC` due to the
complexity of verifying that support. However, this is a good next logical
item to verify.
The `hdtr_positive*` testcases work to a certain degree (the header
testcases pass), but the trailer testcases do not work (it is an expected
failure). In particular, the value received by the mock server doesn't match
the expected value, and instead looks something like the following (using
python array notation):
`trailer[:]message[1:]`
instead of:
`message[:]trailer[:]`
This makes me think there's a buffer overrun issue or problem with the
offset somewhere in the sendfile(2) system call, but I need to do some
other testing first to verify that the code is indeed sane, and my
assumptions/code isn't buggy.
The `sbytes_negative` testcases that check `sbytes` being set to an
invalid value resulting in `EFAULT` fails today as the other change
(which checks `copyout(9)`) has not been committed [1]. Thus, it
should remain an expected failure (see bug 232210 for more details
on this item).
Next steps for testing sendfile(2):
1. Fix the header/trailer testcases so that they pass.
2. Setup if_tap interface and test with it, instead of using "localhost", per
@asomers's suggestion.
3. Handle short recv(2)'s in `server_cat(..)`.
4. Add `SF_SYNC` support.
5. Add some more negative tests outside the scope of the functional contract.
MFC after: 1 month
Reviewed by: asomers
Approved by: emaste (mentor)
PR: 232210
Sponsored by: Netflix, Inc
Differential Revision: https://reviews.freebsd.org/D18625
