Some points for the future:
- libc is not the right place for sorting algorithms.
Probably libutil is better suited for this purpose or
a dedicated libsort. Should move all sorting algorithms
away from libc eventually.
- CheriBSD uses capabilities for memory access, and could
benefit from a standard memswap() function.
- Do something about qsort() in FreeBSD's libc like:
- Mark it deprecated on FreeBSD, as a first step,
due to missing limits on CPU time.
- Audit the use of qsort() in the FreeBSD base system
and consider swapping to other existing sorting
algorithms.
Discussed with: brooks@
Differential Revision: https://reviews.freebsd.org/D36493
This reverts commit a7469c9c0a.
This reverts commit 7d65a450cd.
This reverts commit 8dcf3a82c5.
Several makefile depend on tools built for host.
At least when using DIRDEPS_BUILD we can build these for the
pseudo machine "host" to facilitate building on older host versions.
Ideally we would build these tools in their own directories to avoid
building more than needed.
For now, setting an appropriate default for BTOOLSPATH will suffice
Reviewed by: stevek
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D39708
If there are less than two elements avoid executing the first
sorting loop. No functional change intended.
Reviewed by: kib@
MFC after: 1 week
Sponsored by: NVIDIA Networking
Differential Revision: https://reviews.freebsd.org/D39691
When sorting, both the C11 standard (ISO/IEC 9899:2011, K.3.6.3.2) and
the ISO/IEC JTC1 SC22 WG14 N1172 standard, does not define objects of
zero size as undefined behaviour. However Microsoft's cpp-docs does.
Add proper checks for this. Found while working on bsort(3).
Reviewed by: kib@ and emaste@
MFC after: 1 week
Sponsored by: NVIDIA Networking
Differential Revision: https://reviews.freebsd.org/D39687
The bsort(3) algorithm works by swapping objects, similarly to qsort(3),
and does not require any significant amount of additional memory.
The bsort(3) algorithm doesn't suffer from the processing time issues
known the plague the qsort(3) family of algorithms, and is bounded by
a complexity of O(log2(N) * log2(N) * N), where N is the number of
elements in the sorting array. The additional complexity compared to
mergesort(3) is a fair tradeoff in situations where no memory may
be allocated.
The bsort(3) APIs are identical to those of qsort(3), allowing for
easy drop-in and testing.
The design of the bsort(3) algorithm allows for future parallell CPU
execution when sorting arrays. The current version of the bsort(3)
algorithm is single threaded. This is possible because fixed areas
of the sorting data is compared at a time, and can easily be divided
among different CPU's to sort large arrays faster.
Reviewed by: gbe@, delphij@, pauamma_gundo.com (manpages)
Sponsored by: NVIDIA Networking
Differential Revision: https://reviews.freebsd.org/D36493
These will hopefully be fixed upstream eventually, but silence the
warnings until then.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D39518
veriexec_check_path_label() can be used to check if a specified
path has a label associated with it that contains the what we
want.
Obtained from: Juniper Networks, Inc.
Ensure veriexec opens the file before doing any read operations.
When the MAC_VERIEXEC_CHECK_PATH_SYSCALL syscall is requested, veriexec
needs to open the file before calling mac_veriexec_check_vp. This is to
ensure any set up is done by the file system. Most file systems do not
explicitly need an open, but some (e.g. virtfs) require initialization
of access tokens (file identifiers, etc.) before doing any read or write
operations.
The evaluate_fingerprint() function needs to ensure it has an open file
for reading in order to evaluate the fingerprint. The ideal solution is
to have a hook after the VOP_OPEN call in vn_open. For now, we open the
file for reading, envaluate the fingerprint, and close the file. While
this leaves a potential hole that could possibly be taken advantage of
by a dedicated aversary, this code path is not typically visited often
in our use cases, as we primarily encounter verified mounts and not
individual files. This should be considered a temporary workaround until
discussions about the post-open hook have concluded and the hook becomes
available.
Add MAC_VERIEXEC_GET_PARAMS_PATH_SYSCALL and
MAC_VERIEXEC_GET_PARAMS_PID_SYSCALL to mac_veriexec_syscall so we can
fetch and check label contents in an unconstrained manner.
Add a check for PRIV_VERIEXEC_CONTROL to do ioctl on /dev/veriexec
Make it clear that trusted process cannot be debugged. Attempts to debug
a trusted process already fail, but the failure path is very obscure.
Add an explicit check for VERIEXEC_TRUSTED in
mac_veriexec_proc_check_debug.
We need mac_veriexec_priv_check to not block PRIV_KMEM_WRITE if
mac_priv_gant() says it is ok.
Reviewed by: sjg
Obtained from: Juniper Networks, Inc.
For a process supervisor using the reaper API to track process subtrees,
it is very useful to know the state of the processes on the list.
Sponsored by: https://www.patreon.com/valpackett
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D39585
Introduce the OpenBSD syntax of "scrub" option for "match" and "pass"
rules and the "set reassemble" flag. The patch is backward-compatible,
pf.conf can be still written in FreeBSD-style.
Obtained from: OpenBSD
MFC after: never
Sponsored by: InnoGames GmbH
Differential Revision: https://reviews.freebsd.org/D38025
FREEBSD32 API is also supported on aarch64. Refer to aarch64 instead of
arm64 because it's about the architecture (powerpc64), not the kernel
interface...
Sponsored by: Netflix
by making it accept some open(2) flags. More precisely, only
O_CLOEXEC is supported, the flag is translated into the KQUEUE_CLOEXEC flag
for kqueuex(2), and O_NONBLOCK is silently ignored.
Reported and tested by: vishwin
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D39377
If a malicious casper process sent a host or service string that was
too long, cap_getnameinfo would overrun the caller's buffer by one byte.
The backends for this function needlessly allocated one extra byte
for these buffers. This was harmless, but could be confusing to readers.
Reported by: Coverity (an internal run at Dell)
Reviewed by: oshogbo, emaste
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D39347
There is no real need to close descriptors before a process exits, but
these close calls demonstrate by example that kqueue descriptors occupy
the same namespace as other file descriptors.
Reviewed by: fernape, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39376
Notable upstream pull request merges:
#12194 Fix short-lived txg caused by autotrim
#13368 ZFS_IOC_COUNT_FILLED does unnecessary txg_wait_synced()
#13392 Implementation of block cloning for ZFS
#13741 SHA2 reworking and API for iterating over multiple implementations
#14282 Sync thread should avoid holding the spa config write lock
when possible
#14283 txg_sync should handle write errors in ZIL
#14359 More adaptive ARC eviction
#14469 Fix NULL pointer dereference in zio_ready()
#14479 zfs redact fails when dnodesize=auto
#14496 improve error message of zfs redact
#14500 Skip memory allocation when compressing holes
#14501 FreeBSD: don't verify recycled vnode for zfs control directory
#14502 partially revert PR 14304 (eee9362a7)
#14509 Fix per-jail zfs.mount_snapshot setting
#14514 Fix data race between zil_commit() and zil_suspend()
#14516 System-wide speculative prefetch limit
#14517 Use rw_tryupgrade() in dmu_bonus_hold_by_dnode()
#14519 Do not hold spa_config in ZIL while blocked on IO
#14523 Move dmu_buf_rele() after dsl_dataset_sync_done()
#14524 Ignore too large stack in case of dsl_deadlist_merge
#14526 Use .section .rodata instead of .rodata on FreeBSD
#14528 ICP: AES-GCM: Refactor gcm_clear_ctx()
#14529 ICP: AES-GCM: Unify gcm_init_ctx() and gmac_init_ctx()
#14532 Handle unexpected errors in zil_lwb_commit() without ASSERT()
#14544 icp: Prevent compilers from optimizing away memset()
in gcm_clear_ctx()
#14546 Revert zfeature_active() to static
#14556 Remove bad kmem_free() oversight from previous zfsdev_state_list
patch
#14563 Optimize the is_l2cacheable functions
#14565 FreeBSD: zfs_znode_alloc: lock the vnode earlier
#14566 FreeBSD: fix false assert in cache_vop_rmdir when replaying ZIL
#14567 spl: Add cmn_err_once() to log a message only on the first call
#14568 Fix incremental receive silently failing for recursive sends
#14569 Restore ASMABI and other Unify work
#14576 Fix detection of IBM Power8 machines (ISA 2.07)
#14577 Better handling for future crypto parameters
#14600 zcommon: Refactor FPU state handling in fletcher4
#14603 Fix prefetching of indirect blocks while destroying
#14633 Fixes in persistent error log
#14639 FreeBSD: Remove extra arc_reduce_target_size() call
#14641 Additional limits on hole reporting
#14649 Drop lying to the compiler in the fletcher4 code
#14652 panic loop when removing slog device
#14653 Update vdev state for spare vdev
#14655 Fix cloning into already dirty dbufs
#14678 Revert "Do not hold spa_config in ZIL while blocked on IO"
Obtained from: OpenZFS
OpenZFS commit: 431083f75b
Get/set commands can now choose to provide the interface name rather
than the interface index. This allows userspace to avoid a call to
if_nametoindex().
Suggested by: melifaro
Reviewed by: melifaro
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D39359
The cap_dns tests require Internet access. Currently they fail when
that's not available, which for CI purposes is undesirable. Let's
instead skip the tests if none of the non-casper name/addr lookups
succeed.
To that end:
- Convert the tests to ATF so that skipping is easier to implement.
- Break up the tests into separate test cases.
- If one of the system (i.e., non-casper) lookup functions fails, skip
the test if all of them failed, otherwise fail the tests, since
partial failure indicates something is flaky and deserves a closer
look.
Reviewed by: oshogbo
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D39241
These tests fail spuriously when run in parallel, since some of them
write a value to a global sysctl, read it back, and expect to see the
written value. These tests complete quite quickly in any case.
MFC after: 1 week
memmem started as a GNU extension but is now widely available.
Reviewed by: mhorne (slightly earlier version)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39337
This does not remove LLVM_TARGET_MIPS. Note that the only
MACHINE_ARCH values ending in 'hf' were all MIPS architectures, hence
removing the pattern matches for 'hf'.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D39331
These libraries are linked to directly by applications rather than
opened at runtime via dlopen().
Discussed with: oshogbo
Reviewed by: markj, emaste
Differential Revision: https://reviews.freebsd.org/D39245
This reverts commit ab80f0b21f. The intent
of this change was to avoid possible compilation errors when certain
.inc files were not regenerated, but the method turns out to cause way
more rebuilds than anticipated. Another method will have to be found,
and in the mean time, WITH_CLEAN is the solution that always works.
Fixes: ab80f0b21f
This flag ensures that the tblgen tools do not actually touch the
produced .inc file, if there are no changes to the contents. In turn,
this may prevent a number of rebuilds of files that include such .inc
files, saving build time.
While here, ensure that the shell invocations to locate the used tblgen
binary do not show unnecessary error messages.
Reported by: des
MFC after: 1 week
There are FAT12 and FAT16 file systems, but FAT13 of was an
unintentional invention of mine ...
Reported by: Ravi Pokala <rpokala@freebsd.org>
MFC after: 1 month
This update implements tallying of free directory entries during
create, delete, or rename operations on FAT12 and FAT16 file systems.
Prior to this change, the total number of root directory entries
was reported as number of inodes, but 0 as the number of free
inodes, causing system health monitoring software to warn about
a suspected disk full issue.
The FAT12 and FAT16 file systems provide a limited number of
root directory entries, e.g. 512 on typical hard disk formats.
The valid range of values is 1 to 65535, but the msdosfs code
will effectively round up "odd" values to the next multiple of 16
(e.g. 513 would allow for 528 root directory entries).
This update implements tracking of directory entries during create,
delete, or rename operations, with initial values determined by
scanning the directory when the file system is mounted.
Total and free directory entries are reported in the f_files and
f_ffree elements of struct statfs, despite differences in semantics
of these values:
- There is no limit on the number of files and directories that can
be created on a FAT file system. Only the root directory of FAT12
and FAT16 file systems is limited, any number of files can still be
created in sub-directories, even when 0 free "inodes" are reported.
- A single file can require 1 to 21 directory entries, depending on
the character set, structure, and length of the name. The DOS 8.3
style file name takes up 1 entry, and if the name does not comply
with the syntax of a DOS 8.3 file name, 1 additional entry is used
for each 13 characters of the file name. Since all these entries
have to be contiguous, it is possible that a file or directory with
a long name can not be created, despite a sufficient total number of
free directory entries.
- Renaming a file can require more directory entries than currently
allocated to store its long name, which may prevent an in-place
update of the name if more entries are needed. This may cause a
rename operation to fail if no contiguous range of free entries for
the new name can be found.
- The volume label is stored in a directory entry. An empty FAT file
system with a volume label will therefore show 1 used "inode" in
df.
- The perceentage of free inodes shown in df or monitoring tools does
only represent the state of the root directory of a FAT12 or FAT16
file system. Neither does a reported value of 0% free inodes does
prevent files from being created in sub-directories, nor does a
value of 50% free inodes guarantee that even a single file with
a "long" name can be created in the root directory (if every other
directory entry is occupied and there are no 2 contiguous entries).
The statfs(2) and df(1) man pages have been updated with a notice
regarding the possibly different semantics of values reported as
total and free inodes for non-Unix file systems.
PR: 270053
Reported by: Ben Woods <woodsb02@freebsd.org>
Approved by: mckusick
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D38987
Fix newlocale() overwriting the locale name in collate object
when same instance was used for those locales, and querylocale()
reporting unexpected value for LC_COLLATE_MASK.
PR: 255646, 269375
Reviewed by: markj, bapt (previous version)
Differential Revision: https://reviews.freebsd.org/D30146