Sync libarchive with vendor.
Relevant vendor changes:
Issue #795: XAR - do not try to add xattrs without an allocated name
PR #812: non-recursive option for extract and list
PR #958: support reading metadata from compressed files
PR #999: add --exclude-vcs option to bsdtar
Issue #1062: treat empty archives with a GNU volume header as valid
PR #1074: Handle ZIP files with trailing 0s in the extra fields
(Android APK archives)
PR #1109: Ignore padding in Zip extra field data (Android APK archives)
PR #1167: fix problems related to unreadable directories
Issue #1168: fix handling of strtol() and strtoul()
PR #1172: RAR5 - fix invalid window buffer read in E8E9 filter
PR #1174: ZIP reader - fix of MSZIP signature parsing
PR #1175: gzip filter - fix reading files larger than 4GB from memory
PR #1177: gzip filter - fix memory leak with repeated header reads
PR #1180: ZIP reader - add support for Info-ZIP Unicode Path Extra Field
PR #1181: RAR5 - fix merge_block() recursion
(OSS-Fuzz 12999, 13029, 13144, 13478, 13490)
PR #1183: fix memory leak when decompressing ZIP files with LZMA
PR #1184: fix RAR5 OSS-Fuzz issues 12466, 14490, 14491, 12817
OSS-Fuzz 12466: RAR5 - fix buffer overflow when parsing huffman tables
OSS-Fuzz 14490, 14491: RAR5 - fix bad shift-left operations
OSS-Fuzz 12817: RAR5 - handle a case with truncated huffman tables
PR #1186: RAR5 - fix invalid type used for dictionary size mask
(OSS-Fuzz 14537)
PR #1187: RAR5 - fix integer overflow (OSS-Fuzz 14555)
PR #1190: RAR5 - RAR5 don't try to unpack entries marked as directories
(OSS-Fuzz 14574)
PR #1196: RAR5 - fix a potential SIGSEGV on 32-bit builds
OSS-Fuzz 2582: RAR - fix use after free if there is an invalid entry
OSS-Fuzz 14331: RAR5 - fix maximum owner name length
OSS-Fuzz 13965: RAR5 - use unsigned int for volume number + range check
Additional RAR5 reader changes:
- support symlinks, hardlinks, file owner, file group, versioned files
- change ARCHIVE_FORMAT_RAR_V5 to 0x100000
- set correct mode for readonly directories
- support readonly, hidden and system Windows file attributes
MFC after: 2 weeks
Historically we have not distinguished between kernel wirings and user
wirings for accounting purposes. User wirings (via mlock(2)) were
subject to a global limit on the number of wired pages, so if large
swaths of physical memory were wired by the kernel, as happens with
the ZFS ARC among other things, the limit could be exceeded, causing
user wirings to fail.
The change adds a new counter, v_user_wire_count, which counts the
number of virtual pages wired by user processes via mlock(2) and
mlockall(2). Only user-wired pages are subject to the system-wide
limit which helps provide some safety against deadlocks. In
particular, while sources of kernel wirings typically support some
backpressure mechanism, there is no way to reclaim user-wired pages
shorting of killing the wiring process. The limit is exported as
vm.max_user_wired, renamed from vm.max_wired, and changed from u_int
to u_long.
The choice to count virtual user-wired pages rather than physical
pages was done for simplicity. There are mechanisms that can cause
user-wired mappings to be destroyed while maintaining a wiring of
the backing physical page; these make it difficult to accurately
track user wirings at the physical page layer.
The change also closes some holes which allowed user wirings to succeed
even when they would cause the system limit to be exceeded. For
instance, mmap() may now fail with ENOMEM in a process that has called
mlockall(MCL_FUTURE) if the new mapping would cause the user wiring
limit to be exceeded.
Note that bhyve -S is subject to the user wiring limit, which defaults
to 1/3 of physical RAM. Users that wish to exceed the limit must tune
vm.max_user_wired.
Reviewed by: kib, ngie (mlock() test changes)
Tested by: pho (earlier version)
MFC after: 45 days
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D19908
This is a prerequisite of unifying kernel zlib instances.
Submitted by: Yoshihiro Ota <ota at j.email.ne.jp>
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D20191
During processing we maintain symbol offsets in the 64-bit s_so array,
and when writing the archive convert to 32-bit if no offsets are greater
than 4GB. However, this was somewhat inefficient as we looped over the
array twice: first, converting to big endian and second, writing each
32-bit value one at a time (and incorrectly so on big-endian platforms).
Instead, when writing a 32-bit archive shuffle convert symbol data to
big endian (as required by the ar format) and shuffle to the beginning
of the allocation at the same time.
Also correct emission of the symbol count on big endian platforms.
Further changes are planned, but this should fix powerpc64.
Reported by: jhibbits, mlinimon
Reviewed by: jhibbits, Gerald Aryeetey (earlier)
Tested by: jhibbits
MFC after: 10 days
MFC with: r346079
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D20007
This change allows the user to once again override the C++ standard, restoring
high-level pre-r345708 behavior.
This also unbreaks building lib/ofed/libibnetdisc/Makefile with a non-C++11
capable compiler, e.g., g++ 4.2.1, as the library supported being built with
older C++ standards.
MFC after: 2 weeks
MFC with: r345708
Reviewed by: emaste
Reported by: jbeich
Differential Revision: https://reviews.freebsd.org/D19895 (as part of a larger change)
Previous spellings of my name (NGie, Ngie) weren't my legal spelling. Use Enji
instead for clarity.
While here, remove "All Rights Reserved" from copyrights I "own".
MFC after: 1 week
This is somewhat more readable than pointer arithmetic. Also remove an
unnecessary cast while here.
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
This is a minor simplification; if we do not have any symbols the empty
symbol table can be in 32-bit format.
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Add a stat counter to track ipv6 atomic fragments. Atomic fragments can be
generated in response to invalid path MTU values, but are also a potential
attack vector and considered harmful (see RFC6946 and RFC8021).
While here add tracking of the atomic fragment counter to netstat and systat.
Reviewed by: tuexen, jtl, bz
Approved by: jtl (mentor), bz (mentor)
Event: Aberdeen hackathon 2019
Differential Revision: https://reviews.freebsd.org/D17511
This allows an endless stream of random data within the given bounds.
It already worked if a seed was provided as the 4th argument but not
if one was left out.
In collaboration with: jhb
MFC after: 2 weeks
Relnotes: yes
Experimentally, reduces sort -R time of a 148160 line corpus from about
3.15s to about 0.93s on this particular system.
There's probably room for improvement using some digest other than md5, but
I don't want to look at sort(1) anymore. Some discussion of other possible
improvements in the Test Plan section of the Differential.
PR: 230792
Reviewed by: jhb (earlier version)
Differential Revision: https://reviews.freebsd.org/D19885
Bound input file processing length to avoid the issue reported in [1]. For
simplicity, only allow regular file and character device inputs. For
character devices, only allow /dev/random (and /dev/urandom symblink).
32 bytes of random is perfectly sufficient to seed MD5; we don't need any
more. Users that want to use large files as seeds are encouraged to truncate
those files down to an appropriate input file via tools like sha256(1).
(This does not change the sort algorithm of sort -R.)
[1]: https://lists.freebsd.org/pipermail/freebsd-hackers/2018-August/053152.html
PR: 230792
Reported by: Ali Abdallah <aliovx AT gmail.com>
Relnotes: yes
There's no reason to order based on strcmp of ASCII digests instead of
memcmp of the raw digests.
While here, remove collision fallback. If you collide two MD5s, they're
probably the same string anyway. If robustness against MD5 collisions is
desired, maybe we shouldn't use MD5.
None of the behavior of sort -R is specified by POSIX, so we're free to
implement this however we like. E.g., using a 128-bit counter and block cipher
to generate unique indices for each line of input.
PR: 230792 (2/many)
Relnotes: This will change the sort order for a given dataset with a
given seed. Other similarly breaking changes are planned.
Sponsored by: Dell EMC Isilon
from 1.0.0:
Add "continuation" flag, to allow multiple "xo" invocations in a single line of output (#58)
Add --top-wrap to make top-level JSON wrappers
Add --{open,close}-{list,instace} options
Add xo_xml_leader(), to detect use of some bogus XML tags. It's still bad form, but it's a little safer now
Avoid call to xo_write before xo_flush, since the latter calls the former
Check return code from xo_flush_h properly (<0) (FreeBSD Bug 236935)
For JSON output, avoid newline before a container's close brace (#62)
Merge branch 'text_only' of https://github.com/zvr/libxo into zvr-text_only
Use XO_USE_INT_RETURN_CODES, not USE_INT_RETURN_CODES
add docs for --continuation
add docs for --not-first
call xo_state_set_flags before values and close containers; add XOIF_MADE_OUTPUT flag to track state; make proper empty JSON objects in xo_finish
color_map code has to be #ifdef'd out, since the struct definition
correct xo_flush_func_t (doesn't use xo_ssize_t)
make depth change for --top-wrap only for JSON
fix to handle --top-wrap in "xo" by being more consistent with handling trailing newlines
fix to handle text-only version #64 (from zvr)
fix xo_buf_has_room for round up to the next XO_BUFSIZ, not just add XO_BUFSIZ to the size (FreeBSD Bug 236937)
update docs for new "xo" options
update functions to use xo_ssize_t
update test cases
from 1.0.1:
Add EINTEGRITY to .pot files under test/gettext/ (fix from FreeBSD)
from 1.0.2:
handle failure from xo_vnsprintf; don't add -1 to "rc"
PR: 236937, 236935
Submitted by: phil
Reported by: Alfonso S. Siciliano <alfix86@gmail.com>
MFC after: 2 weeks
The value was changed from int to bool. Since the new type
is smaller, the rest of the variable in the caller was left
unitialized.
PR: 236714
Reported by: trasz
Diagnosed by: markj
Sponsored by: The FreeBSD Foundation
to no longer be displayed. This was because the reimplementation of
setup_buffer() did not copy the previous contents into any reallocated
buffer.
Reported by: James Wright <james.wright@jigsawdezign.com>
PR: 236947
MFC after: 3 days
CXXSTD was added as the C++ analogue to CSTD.
CXXSTD defaults to `-std=c++11` with supporting compilers; `-std=gnu++98`,
otherwise for older versions of g++.
This change standardizes the CXXSTD variable, originally added to
googletest.test.inc.mk as part of r345203.
As part of this effort, convert all `CXXFLAGS+= -std=*` calls to use `CXXSTD`.
Notes:
This value is not sanity checked in bsd.sys.mk, however, given the two
most used C++ compilers on FreeBSD (clang++ and g++) support both modes, it is
likely to work with both toolchains. This method will be refined in the future
to support more variants of C++, as not all versions of clang++ and g++ (for
instance) support C++14, C++17, etc.
Any manual appending of `-std=*` to `CXXFLAGS` should be replaced with CXXSTD.
Example:
Before this commit:
```
CXXFLAGS+= -std=c++14
```
After this commit:
```
CXXSTD= c++14
```
Reviewed by: asomers
Approved by: emaste (mentor)
MFC after: 1 month
MFC with: r345203, r345704, r345705
Relnotes: yes
Tested with: make tinderbox
Differential Revision: https://reviews.freebsd.org/D19732
When a review is closed via Phabricator it updates the patch attached to the
review. I downloaded the raw patch from Phabricator, applied it, and repeated
my mistake from r345704 by accident mixing content from D19732 and D19738.
For my own personal sanity, I will try not to mix reviews like this in the
future.
MFC after: 1 month
MFC with: r345706
Approved by: emaste (mentor, implicit)
CXXSTD was added as the C++ analogue to CSTD.
CXXSTD defaults to `-std=c++11` with supporting compilers; `-std=gnu++98`,
otherwise for older versions of g++.
This change standardizes the CXXSTD variable, originally added to
googletest.test.inc.mk as part of r345203.
As part of this effort, convert all `CXXFLAGS+= -std=*` calls to use `CXXSTD`.
Notes:
This value is not sanity checked in bsd.sys.mk, however, given the two
most used C++ compilers on FreeBSD (clang++ and g++) support both modes, it is
likely to work with both toolchains. This method will be refined in the future
to support more variants of C++, as not all versions of clang++ and g++ (for
instance) support C++14, C++17, etc.
Any manual appending of `-std=*` to `CXXFLAGS` should be replaced with CXXSTD.
Example:
Before this commit:
```
CXXFLAGS+= -std=c++14
```
After this commit:
```
CXXSTD= c++14
```
Reviewed by: asomers
Approved by: emaste (mentor)
MFC after: 1 month
MFC with: r345203, r345704, r345705
Relnotes: yes
Tested with: make tinderbox
Differential Revision: https://reviews.freebsd.org/D19732
I accidentally committed code from two reviews. I will reintroduce the code to
bsd.progs.mk as part of a separate commit from r345704.
Approved by: emaste (mentor, implicit)
MFC after: 2 months
MFC with: r345704
CXXSTD defaults to `-std=c++11` with supporting compilers; `-std=gnu++98`,
otherwise for older versions of g++.
This change standardizes the CXXSTD variable, originally added to
googletest.test.inc.mk as part of r345203.
As part of this effort, convert all `CXXFLAGS+= -std=*` calls to use `CXXSTD`.
Notes:
This value is not sanity checked in bsd.sys.mk, however, given the two
most used C++ compilers on FreeBSD (clang++ and g++) support both modes, it is
likely to work with both toolchains. This method will be refined in the future
to support more variants of C++, as not all versions of clang++ and g++ (for
instance) support C++14, C++17, etc.
Any manual appending of `-std=*` to `CXXFLAGS` should be replaced with CXXSTD.
Example:
Before this commit:
```
CXXFLAGS+= -std=c++14
```
After this commit:
```
CXXSTD= c++14
```
Reviewed by: asomers
Approved by: emaste (mentor)
MFC after: 1 month
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D19732
Highlights:
- Bugfix for order in which /delete-node/ and /delete-property/ are
processed [0]
- /omit-if-no-ref/ support has been added (used only by U-Boot at this
point, in theory)
- GPL dtc compat version bumped to 1.4.7
- Various small fixes and compatibility improvements
Reported by: strejda [0]
MFC after: 1 week
exist
Apply EX_UNAVAILABLE patch part from PR 170775 to match the documentation.
Checked with a command from PR 210770:
lockf -n /tmp/doesnotexist echo; echo $?
PR: 210770
MFC after: 1 week
- Fix markup.
- Mention that process can only allow tracing for itself. This is already
stated in procctl(2), but requiring knowledge of the syscall description
is too much for the tool user.
- Clearly state that query mode only works for existing process.
Noted and reviewed by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 3 days
r344504 added an extra ARP_LOG() call in case of an if_output() failure.
It turns out IPv4 can be noisy. In order to not spam the console by default:
(a) add a counter for these events so people can keep better track of how
often it happens, and
(b) add a sysctl to select the default ARP_LOG log level and set it to
INFO avoiding the one (the new) DEBUG level by default.
Claim a spare (1st one after 10 years since the stats were added) in order
to not break netstat from FreeBSD 12->13 updates in the future.
Reviewed by: karels
Differential Revision: https://reviews.freebsd.org/D19490
This matches GNU seq, for example.
For users that are looking for similar functionality, 'jot -b foo N' will
print 'foo' N times. See jot(1).
PR: 236347
Reported by: <y AT maya.st>
Sponsored by: Dell EMC Isilon
