import bsd diff3 from OpenBSD.
Differences with OpenBSD:
- lots of warning fixed
- no shell wrapper with diff3 actually living in libexec
- capsicumized
Keep it disconnected as it is not yet good enough to replace GNU diff
The motivation to import it now it to allow other people to jump in and also to
have an open development on it
Obtained from: OpenBSD
Decoding of the third argument depends on the first one. For doing this,
add a corresponding function to libsysdecode.
Thanks to jhb@ for suggesting this.
Extend the ino_t, dev_t, nlink_t types to 64-bit ints. Modify
struct dirent layout to add d_off, increase the size of d_fileno
to 64-bits, increase the size of d_namlen to 16-bits, and change
the required alignment. Increase struct statfs f_mntfromname[] and
f_mntonname[] array length MNAMELEN to 1024.
ABI breakage is mitigated by providing compatibility using versioned
symbols, ingenious use of the existing padding in structures, and
by employing other tricks. Unfortunately, not everything can be
fixed, especially outside the base system. For instance, third-party
APIs which pass struct stat around are broken in backward and
forward incompatible ways.
Kinfo sysctl MIBs ABI is changed in backward-compatible way, but
there is no general mechanism to handle other sysctl MIBS which
return structures where the layout has changed. It was considered
that the breakage is either in the management interfaces, where we
usually allow ABI slip, or is not important.
Struct xvnode changed layout, no compat shims are provided.
For struct xtty, dev_t tty device member was reduced to uint32_t.
It was decided that keeping ABI compat in this case is more useful
than reporting 64-bit dev_t, for the sake of pstat.
Update note: strictly follow the instructions in UPDATING. Build
and install the new kernel with COMPAT_FREEBSD11 option enabled,
then reboot, and only then install new world.
Credits: The 64-bit inode project, also known as ino64, started life
many years ago as a project by Gleb Kurtsou (gleb). Kirk McKusick
(mckusick) then picked up and updated the patch, and acted as a
flag-waver. Feedback, suggestions, and discussions were carried
by Ed Maste (emaste), John Baldwin (jhb), Jilles Tjoelker (jilles),
and Rick Macklem (rmacklem). Kris Moore (kris) performed an initial
ports investigation followed by an exp-run by Antoine Brodin (antoine).
Essential and all-embracing testing was done by Peter Holm (pho).
The heavy lifting of coordinating all these efforts and bringing the
project to completion were done by Konstantin Belousov (kib).
Sponsored by: The FreeBSD Foundation (emaste, kib)
Differential revision: https://reviews.freebsd.org/D10439
catman(1) checks if mandoc(1) do support the manpage before trying to generate
the catpage and falls back on nroff, using the same mechanism as man(1).
Per POSIX, allow passing multiple pathnames to uncompress -c, concatenating
the uncompressed data.
Passing multiple pathnames to compress -c remains disallowed, since the
result cannot be decompressed.
PR: 219387
Reported by: Jörg Schilling
Metadata printing with -b, -H, or -n flags suffered from a few flaws:
1) -b/offset printing was broken when used in conjunction with -o
2) With -o, bsdgrep did not print metadata for every match/line, just
the first match of a line
3) There were no tests for this
Address these issues by outputting this data per-match if the -o flag is
specified, and prior to outputting any matches if -o but not --color,
since --color alone will not generate a new line of output for every
iteration over the matches.
To correct -b output, fudge the line offset as we're printing matches.
While here, make sure we're using grep_printline in -A context. Context
printing should *never* look at the parsing context, just the line.
The tests included do not pass with gnugrep in base due to it exhibiting
similar quirky behavior that bsdgrep previously exhibited.
Submitted by: Kyle Evans <kevans91@ksu.edu>
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D10580
We should not set an arbitrary cap on the number of matches on a line,
and in any case MAX_LINE_MATCHES of 32 is much too low. Instead, if we
match more than MAX_LINE_MATCHES, keep processing and matching from the
last match until all are found.
For the regression test, we produce 4096 matches (larger than we expect
we'll ever set MAX_LINE_MATCHES) and make sure we actually get 4096
lines of output with the -o flag.
We'll also make sure that every distinct line is getting its own line
number to detect line metadata not being printed as appropriate along
the way.
PR: 218811
Submitted by: Kyle Evans <kevans91@ksu.edu>
Reported by: jbeich
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D10577
r313948 partially fixed --mmap behavior but was incomplete. This commit
generally reverts it and does it the more correct way- by just consuming
the rest of the buffer and moving on.
PR: 219402
Submitted by: Kyle Evans <kevans91@ksu.edu>
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D10820
This was done by Romain Tartière for PR123553. I initially thought that it would break code like this:
#define b00101010 -1
if (0 b00101010)
...
by joining 0 and b00101010 together. However, the real problem with that patch was that once it saw a 0, it assumed that the number was base 2, 8 or 16, ignoring base 10 floating point numbers. I fixed that.
I didn't copy the diagnostic part of the original patch as it seems out of scope of implementing binary integer literals formatting.
PR: 123553
Submitted by: romain (original version)
Approved by: pfg (mentor)
Items tested via this commit are:
- Some basic POSIX constants.
- Some valid programming environments with -v.
- Some invalid programming environments via -v.
NOTE: this test makes assumptions about ILP32/LP32 vs LP64 that are
currently not true on all architectures to avoid hardcoding some
architectures in the tests. I'm working on improving getconf(1) to be
more sane about handling ILP32/LP32 vs LP64. Future commits are coming
soon to address this.
MFC after: 2 weeks
Tested with: amd64, i386
Sponsored by: Dell EMC Isilon
These tests query a running process for information related to the -b,
-c, -e, and -f flags; the -f testcase is largely stubbed out, pending
additional work to determine a good, deterministic descriptor.
Core file test support is coming soon--it requires a bit more effort
due to the fact that:
- coredumps can be disabled (kern.coredump=0).
- corefiles can be put in different directories than the current
directory, or be named something other than `<prog>.core`
(`kern.corefile`).
MFC after: 2 months
Sponsored by: Dell EMC Isilon
Bring in some bits from NetBSD and lift the restriction in uniq(1) that
-c cannot be used with the -d and -u options. This restriction seems
unnecessary and is supported at least by GNU, OpenBSD, and NetBSD. Lift
the restriction and simplify the show() logic a little bit to maintain
functionality when -c is provided with -d/-u.
Also with this change, -d and -u are now actually a mutually exclusive,
albeit valid, combination. Given that they both indicate opposite
behavior, uniq(1) will no longer output anything if both -d and -u are
supplied. This is in line with NetBSD as well as GNU.
Adjust the man page and usage() to reflect that -c is its own standalone
option.
PR: 200553
Submitted by: Kyle Evans <kevans91@ksu.edu>
Reviewed by: cem, emaste
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D10694
Previously, when given a negative -A/-B/-C argument bsdgrep would
overflow the respective context flag(s) and exhibited surprising
behavior.
Fix this by removing unsignedness of Aflag/Bflag and erroring out if
we're given a value < 0. Also adjust the type used to track 'tail'
context in procfile() so that it accurately reflects the Aflag value
rather than overflowing and losing trailing context.
This also fixes an inconsistency previously existing between -n and
-C "n" behavior. They are now both limited to LLONG_MAX, to be
consistent.
Add some test cases to make sure grep errors out properly for both
negative context values as well as non-numeric context values rather
than giving bogus matches.
Submitted by: Kyle Evans <kevans91@ksu.edu>
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D10675
The _Noreturn attribute was added to placate Coverity and other static
analysis tools. The __printflike attribute was added to catch issues
with the calls related to printf(3) abuse.
- Modify the code to facilitate the __printflike attribute addition.
- Convert errf calls in to_mb(..) and to_mb_string(..) to warn(..) so
the calls will return instead of exiting, as the code suggests it
should.
Differential Revision: D10704
MFC after: 1 month
Reviewed by: pfg
Sponsored by: Dell EMC Isilon
lib/libmt/mtlib.c:
Add the LTO-8 density code to the density table in libmt.
usr.bin/mt/mt.1:
Add the LTO-8 density code, tracks, bpmm, and bpi to the density
table in the mt(1) man page.
MFC after: 3 days
Sponsored by: Spectra Logic
- Fix a typo (SIGIGN -> SIG_IGN). Use .Dv when referencing SIG_IGN.
- Use semi-colons as soft breaks when separating sentences for
the FLAGS section.
- Tweak wording for C slightly to flow better and to be a bit
more technically correct (signals with handlers installed will
be caught by the target program).
- Reference signal(3) in the SEE ALSO section.
MFC after: 3 weeks
Sponsored by: Dell EMC Isilon
that setting the kernels' idea of terminal size is somehow an
alternative to environment variables.
Reported by: kib
MFC after: 2 weeks
Sponsored by: DARPA, AFRL
Add a -C option to specify a maximum capacity for the final image file.
It is useful to control the size of the generated image for sdcard or
when we will add dynamic size partition.
Add --capacity which is a shorthand to define min and max capacity at
the same time.
Reviewed by: bapt, marcel, wblock (manpages)
Sponsored by: Gandi.net
Differential Revision: https://reviews.freebsd.org/D10509
When the input to csplit contains fewer lines than the number of matches
specified, extra output was mistakenly included in some output files.
Fix the bug and add a simple ATF regression test.
PR: 219024
Submitted by: J.R. Oldroyd <fbsd at opal.com>
size is already set to something other than zero. It's supposed to be
called from eg /etc/profile - it's not neccessary to query terminal
size when logging in over the network, because the protocol used already
takes care of this, but it's neccessary when logging over a serial line.
Reviewed by: cem, Daniel O'Connor <darius@dons.net.au>
MFC after: 2 weeks
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10637
for "unable to parse response" error which happens when youre typing
too fast for the machine you're running it on.
Reviewed by: cem, Daniel O'Connor <darius@dons.net.au>
MFC after: 2 weeks
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10624
Among new things it is now threaded by default, use zstd -T# to chose the
number of threads not that -T0 will automatically determine the number of
threads based on the number of CPU online.
This will help application developers simulate end of tape conditions.
To inject an error in sa0:
sysctl kern.cam.sa.0.inject_eom=1
This will return the next read or write request queued with 0 bytes
written. Any subsequent writes or reads will go along as usual.
This will also cause the early warning position flag to get set
for the next position query. So, 'mt status' will show the BPEW
(Beyond Programmable Early Warning) flag on the first query after
an error injection. After that, the position flags will be as they
are in the underlying tape drive.
Also, update the sa(4) man page to describe tape parameters,
which can be set via 'mt param'.
sys/cam/scsi/scsi_sa.c:
In saregister(), create the inject_eom sysctl variable.
In sastart(), check to see whether inject_eom is set. If
so, return the read or write with 0 bytes written to
indicate EOM. Set the set_pews_status flag so that we
fake PEWS status in the next position call for reads, and the
next 3 calls for writes. This allows the user to see the BPEW
flag one time via 'mt status'.
In sagetpos(), check the set_pews_status flag and fake
PEWS status and decrement the counter if it is set.
share/man/man4/sa.4:
Document the inject_eom sysctl variable.
Document all of the parameters currently supported via
'mt param'.
usr.bin/mt/mt.1:
Point the user to the sa(4) man page for more details on
supported parameters.
MFC after: 3 days
Sponsored by: Spectra Logic
Refactoring done in r317703 broke -c, -l, and -L flags implying
suppression of match printing. Fortunately this is just a matter of not
doing any printing of the resulting matches and context printing was not
broken in this refactoring.
Add some regression tests since this area may still see further
refactoring, include different context flags as well even though they
were not broken in this case.
PR: 219077
Submitted by: Kyle kevans91@ksu.edu
Reported by: markj
Reviewed by: cem, ngie
Differential Revision: https://reviews.freebsd.org/D10607
As of r295638, fputs() returns the number of bytes written (if not more than
INT_MAX). This broke csplit completely, since csplit assumed only success
only for the return value 0.
PR: 213510
Submitted by: J.R. Oldroyd
MFC after: 1 week
Relnotes: yes
In BSD grep, fix escape map building in the regex parser. It was
previously using memory not explicitly initialized, and the MBS escape
map was being built based on a version of the pattern with escapes
already parsed out.
This is Kyle's change, but I restored the broken style that already
exists in this file.
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reviewed by: cem, Kyle Evans (my style changes)
Differential Revision: https://reviews.freebsd.org/D10098
-w flag matching with an empty pattern was generally 'broken', allowing
matches to occur on any line whether or not it actually matches -w
criteria.
This fix required a good amount of refactoring to address. procline()
is altered to *only* process the line and return whether it was a match
or not, necessary to be able to short-circuit the whole function in case
of this matchall flag. -m flag handling is moved out as well because it
suffers from the same fate as context handling if we bypass any actual
pattern matching.
The matching context (matches, mostly) didn't previously exist outside
of procline(), so we go ahead and create context object for file
processing bits to pass around. grep_printline() was created due to
this, for the scenarios where the matches don't actually matter and we
just want to print a line or two, a la flushing the context queue and
no -o or --color specified.
Damage from this broken behavior would have been mitigated by the fact
that it is unlikely users would invoke grep -w with an empty pattern.
This was identified while checking PR 105221 for problems it this may
cause in BSD grep, but PR 105221 is *not* a report of this behavior.
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Differential Revision: https://reviews.freebsd.org/D10433
r317049 added -z/--null-data to BSD grep but missed the update to nls
catalogs.
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Differential Revision: https://reviews.freebsd.org/D10456
As reported in r218614 it's useful to have an indication of whether or not
BSD grep was built with GNU_GREP_COMPAT.
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reported by: mandree
Differential Revision: https://reviews.freebsd.org/D10451
-w and -v flag matching was mostly functional but had some minor
problems:
1. -w flag processing only allowed one iteration through pattern
matching on a line. This was problematic if one pattern could match
more than once, or if there were multiple patterns and the earliest/
longest match was not the most ideal, and
2. Previous work "fixed" things to not further process a line if the
first iteration through patterns produced no matches. This is clearly
wrong if we're dealing with the more restrictive -w matching.
#2 breakage could have also occurred before recent broad rewrites, but
it would be more arbitrary based on input patterns as to whether or not
it actually affected things.
Fix both of these by forcing a retry of the patterns after advancing
just past the start of the first match if we're doing more restrictive
-w matching and we didn't get any hits to start with. Also move -v flag
processing outside of the loop so that we have a greater change to match
in the more restrictive cases. This wasn't strictly wrong, but it could
be a little more error prone.
While here, introduce some regressions tests for this behavior and fix
some excessive wrapping nearby that hindered readability. GNU grep
passes these new tests.
PR: 218467, 218811
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reviewed by: cem, ngie
Differential Revision: https://reviews.freebsd.org/D10329
These are only built as part of the top-level 'build-tools' call for
'make buildworld'. They still need to be cleaned during the 'make clean'
treewalks though.
Reported by: markj
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Long ago, sh used to have its own optimized and restricted string formatting
implementation, which the printf builtin had to bypass via asprintf() to a
temporary buffer. Since sh has used libc's string formatting implementation
for a long time, remove the workaround.
Add a check to keep printf %c '' working the same way (output nothing);
POSIX allows both outputting nothing and outputting a NUL byte.
Also, this change avoids silently discarding format directives for whose
output asprintf() cannot allocate memory.
usr.bin/diff/diffreg.c: In function 'change':
usr.bin/diff/diffreg.c:1085: warning: 'i' may be used uninitialized in this function
This version of gcc is not smart enough to see that 'i' cannot actually
be used unitialized. However, the variable is confusingly re-used, so
it is better to give it another name, and clearly initialize it before
attempting to use it.
Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D10484
in the file by adding the global (g) option at the end. Without it, only the
first match is replaced, subsequent ones are ignored. The intention of the
example is to demonstrate something else, but adding the g matches the example
to what the description says.
Discussed with: brd (on IRC)
MFC after: 1 week
Bugs have been found in the fastmatch implementation as used in bsdgrep.
Some have been fixed (r316495) while fixes for others are in review
(D10098).
In comparison with the fastmatch implementation, Kyle Evans found that:
- regex(3)'s performance with literal expressions offers a speed
improvement over fastmatch
- regex(3)'s performance, both with simple BREs and EREs, seems to be
comparable
The regex implementation was imported in r226035, and the commit message
reports:
This is a temporary solution until the whole regex library is
not replaced so that BSD grep development can continue and the
backported code gets some review and testing. This change only
improves scalability slightly, there is no big performance boost
yet but several minor bugs have been found and fixed.
Introduce a WITH_/WITHOUT_BSD_GREP_FASTMATCH knob to support testing
of both approaches.
PR: 175314, 194823
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reviewed by: bdrewery (in part)
Differential Revision: https://reviews.freebsd.org/D10282
etcupdate(8) requires that option, while GNU diff supports many more variation
of that options, their behaviour beside the simple verion implemented here are
quite inconsistent as such I do not plan to implement those.
The only special keyword supported by this implementation are: %< and %>
%= is not implemented as the documentation of GNU diff says: common lines, but
it actually when tested print the changes from the first file
in place. To do per-cpu stats, convert all fields that previously were
maintained in the vmmeters that sit in pcpus to counter(9).
- Since some vmmeter stats may be touched at very early stages of boot,
before we have set up UMA and we can do counter_u64_alloc(), provide an
early counter mechanism:
o Leave one spare uint64_t in struct pcpu, named pc_early_dummy_counter.
o Point counter(9) fields of vmmeter to pcpu[0].pc_early_dummy_counter,
so that at early stages of boot, before counters are allocated we already
point to a counter that can be safely written to.
o For sparc64 that required a whole dummy pcpu[MAXCPU] array.
Further related changes:
- Don't include vmmeter.h into pcpu.h.
- vm.stats.vm.v_swappgsout and vm.stats.vm.v_swappgsin changed to 64-bit,
to match kernel representation.
- struct vmmeter hidden under _KERNEL, and only vmstat(1) is an exclusion.
This is based on benno@'s 4-year old patch:
https://lists.freebsd.org/pipermail/freebsd-arch/2013-July/014471.html
Reviewed by: kib, gallatin, marius, lidl
Differential Revision: https://reviews.freebsd.org/D10156
- Report missing includes at the correct location.
- Add initial support for the -@ option emitting a symbol table.
- Add support for running tests with and without -@
- Add support for generating __fixups__ and __local_fixups__
- Attach the to-string transform to the node path.
r316477 broke zero-length matches when not using the -o flag, by
skipping over them entirely.
Add a regression test so that it doesn't break again in the future.
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reviewed by: cem emaste ngie
Differential Revision: https://reviews.freebsd.org/D10333
Make bsdgrep more sensitive to context overlaps. If it's printing
context that either overlaps or is immediately adjacent to another bit
of context, don't print a separator.
- Non-overlapping segments no longer have two separators between them
- Overlapping segments no longer have separators between them with
overlapping sections repeated
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D10105
This is more sensible than the previous behaviour of grepping stdin,
and matches newer GNU grep behaviour.
PR: 216307
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reviewed by: cem, emaste, ngie
Relnotes: Yes
Differential Revision: https://reviews.freebsd.org/
-z treats input and output data as sequences of lines terminated by a
zero byte instead of a newline. This brings it more in line with GNU grep
and brings us closer to passing the current tests with BSD grep.
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reviewed by: cem
Relnotes: Yes
Differential Revision: https://reviews.freebsd.org/D10101
messages before accessing message fields that may not be present,
removing dead/duplicate/misleading code along the way.
Document the message format for each routing socket message in
route.h.
Fix a bug in usr.bin/netstat introduced in r287351 that resulted in
pointer computation with essentially random 16-bit offsets and
dereferencing of the results.
Reviewed by: ae
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D10330
The destination buffer is sized as the sum of program argument lengths, so
it has plenty of room for *argv. Appease Coverity by using strlcpy instead
of strcpy. Similar to a nearby cleanup performed in r316500.
No functional change.
Reported by: Coverity (CWE-120)
CID: 1006703
Sponsored by: Dell EMC Isilon
Create additional tests to cover regressions that were discovered by
PRs linked to reviews D10098, D10102, and D10104.
It is worth noting that neither bsdgrep(1) nor gnugrep(1) in the base
system currently pass all of these tests, and gnugrep(1) not quite being
up to snuff was also noted in at least one of the PRs.
PR: 175314 202022 195763 180990 197555 197531 181263 209116
Submitted by: Kyle Evans <kevans91@ksu.edu>
Reviewed by: cem, ngie, emaste
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D10112
Invalid expressions with an ultimate compiled pattern length of 0 (e.g.,
"grep -E {") were not taken into account and caused a segfault while trying
to fill in the good suffix table.
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reviewed by: me
Differential Revision: https://reviews.freebsd.org/D10113
xmalloc was a debug malloc implementation, but the x{malloc,calloc,free}
functions default to calling the malloc(3) equivalents.
Instead of relying on this malloc shim, we can devise better ways to debug
malloc issues that aren't misleading upon initial inspection. (I.e., using
jemalloc's various built-in debugging capabilities.)
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reviewed by: emaste, cem
Differential Revision: https://reviews.freebsd.org/D10269
r316477 changed the color output to match exactly the in-tree GNU grep,
but introduces unnecessary escape sequences.
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reported by: ache
MFC after: 1 month
MFC with: r316477
Create a convenience rgrep link for bsdgrep(1) that observes 'grep -r'
behavior.
A follow-up to r316473.
Submitted by: Kyle Evans <kevans91 at ksu.edu>
Reviewed by: emaste (earlier version), cem
Differential Revision: https://reviews.freebsd.org/D10109
- Set REG_NOTBOL if we've already matched beginning of line and we're
examining later parts
- For each pattern we examine, apply it to the remaining bits of the
line rather than (potentially) smaller subsets
- Check for REG_NOSUB after we've looked at all patterns initially
matching the line
- Keep track of the last match we made to later determine if we're
simply not matching any longer or if we need to proceed another byte
because we hit a zero-length match
- Match the earliest and longest bit of each line before moving the
beginning of what we match to further in the line, past the end of the
longest match; this generally matches how gnugrep(1) seems to behave,
and seems like pretty good behavior to me
- Finally, bail out of printing any matches if we were set to print all
(empty pattern) but -o (output matches) was set
PR: 195763, 180990, 197555, 197531, 181263, 209116
Submitted by: "Kyle Evans" <kevans91@ksu.edu>
Reviewed by: cem
MFC after: 1 month
Relnotes: Yes
Differential Revision: https://reviews.freebsd.org/D10104
top(1) read the wrong amount of data from sysctl, uint64_t instead of
boolean_t, resulting in the stats not showing in many cases.
X-MFC-With: r315435
Sponsored by: ScaleEngine Inc.
resulting in a process dumping core in the corefile.
Also extend procstat to view select members of 'struct ptrace_lwpinfo'
from the contents of the note.
Sponsored by: Dell EMC Isilon
Show nanoseconds in the -u/-c header line.
The present portability conditionals cannot handle the POSIX standard
st_mtim, so remove them and unconditionally use st_mtim.
PR: 218018
Reported by: jbeich
Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D10145
implemented in top(1), rather than relying on K&R prototypes, which can
cause problems on targets where there are multiple incompatible calling
conventions and the compiler requires argument information to select the
correct one.
(There's a bit more to do here, since it looks like top(1) also sometimes
provides prototypes for various curses functions rather than relying on
the header file...)
Sponsored by: DARPA, AFRL
MFC after: 1 week
Sync libarchive with vendor.
Vendor changes (FreeBSD-related):
- store extended attributes with extattr_set_link() if no fd is provided
- add extended attribute tests to libarchive and bsdtar
- fix tar's test_option_acls
- support the UF_HIDDEN file flag
X-MFC with: 315636
naming scheme
usr.bin/diff/diff_test was renamed to usr.bin/diff/netbsd_diff_test
to avoid collisions with the renamed FreeBSD test.
MFC after: 1 week
Sponsored by: Dell EMC Isilon
This is a painful change, but it is needed. On the one hand, we avoid
modifying them, and this slows down some ideas, on the other hand we still
eventually modify them and tools like netstat(1) never work on next version of
FreeBSD. We maintain a ton of spares in them, and we already got some ifdef
hell at the end of tcpcb.
Details:
- Hide struct inpcb, struct tcpcb under _KERNEL || _WANT_FOO.
- Make struct xinpcb, struct xtcpcb pure API structures, not including
kernel structures inpcb and tcpcb inside. Export into these structures
the fields from inpcb and tcpcb that are known to be used, and put there
a ton of spare space.
- Make kernel and userland utilities compilable after these changes.
- Bump __FreeBSD_version.
Reviewed by: rrs, gnn
Differential Revision: D10018
Decode the last argument to ioctl() as a pointer rather than an int.
Eventually this could use 'int' for the _IOWINT() case and pointers for
all others.
The last argument to sendto() is a socklen_t value, not a pointer.
Provides:
amount of compressed data
logical size of compressed data (how much it would have taken uncompressed)
compression ratio (logical size : total ARC size)
Overhead (space consumed for compression headers)
Example output:
ARC: 31G Total, 18G MFU, 9067M MRU, 2236K Anon, 615M Header, 2947M Other
25G Compressed, 54G Uncompressed, 1.76:1 Ratio, 2265M Overhead
Reviewed by: jpaetzel, smh, imp, jhb (previous version)
MFC after: 2 week
Relnotes: yes
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D9829
Previously, the offset in a system call description specified the
array index of the start of a system call argument. For most system
call arguments this was the same as the index of the argument in the
function signature. 64-bit arguments (off_t and id_t values) passed
on 32-bit platforms use two slots in the array however. This was
handled by adding (QUAD_SLOTS - 1) to the slot indicies of any
subsequent arguments after a 64-bit argument (though written as ("{
Quad, 1 }, { Int, 1 + QUAD_SLOTS }" rather than "{ Quad, 1 }, { Int, 2
+ QUAD_SLOTS - 1 }"). If a system call contained multiple 64-bit
arguments (such as posix_fadvise()), then additional arguments would
need to use 'QUAD_SLOTS * 2' but remember to subtract 2 from the
initial number, etc. In addition, 32-bit powerpc requires 64-bit
arguments to be 64-bit aligned, so if the effective index in the array
of a 64-bit argument is odd, it needs QUAD_ALIGN added to the current
and any subsequent slots. However, if the effective index in the
array of a 64-bit argument was even, QUAD_ALIGN was omitted.
This approach was messy and error prone. This commit replaces it with
automated pre-processing of the system call table to do fixups for
64-bit argument offsets. The offset in a system call description now
indicates the index of an argument in the associated function call's
signature. A fixup function is run against each decoded system call
description during startup on 32-bit platforms. The fixup function
maintains an 'offset' value which holds an offset to be added to each
remaining system call argument's index. Initially offset is 0. When
a 64-bit system call argument is encountered, the offset is first
aligned to a 64-bit boundary (only on powerpc) and then incremented to
account for the second argument slot used by the argument. This
modified 'offset' is then applied to any remaining arguments. This
approach does require a few things that were not previously required:
1) Each system call description must now list arguments in ascending
order (existing ones all do) without using duplicate slots in the
register array. A new assert() should catch any future
descriptions which violate this rule.
2) A system call description is still permitted to omit arguments
(though none currently do), but if the call accepts 64-bit
arguments those cannot be omitted or incorrect results will be
displated on 32-bit systems.
Tested on: amd64 and i386
.../usr.bin/diff/diff_test
Some minor adjustment needed to be done for :same as it currently
has the test script hardcoded into the test, instead of using an
idiom like $(dirname $0)
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Prefer ${SRCTOP}/foo over ${.CURDIR}/../../foo and ${SRCTOP}/usr.bin/foo
over ${.CURDIR}/../foo for paths in Makefiles.
Differential Revision: https://reviews.freebsd.org/D9932
Sponsored by: Netflix
Silence on: arch@ (twice)
longer a special case.
- Prefer PREFIX/share/man over PREFIX/man.
- Add /usr/local/share/man to man_default_path.
- Update manpath man page.
Reviewed by: bapt
The additional testcases use absolute paths for sources and targets,
as the other testcase which tested `-l sr` used flat relative paths in
the same directory.
Please note that these testcases do not test `-l a` -- that's already
addressed in the battery of tests.
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Unfortunately kyua does not omit the path mismatch on failure, so it must be coded
into the error message.
Cache the values, run the test(1) call, then print out the values in an atf_fail
call to emit the required diagnostics to debug why things are failing.
MFC after: 1 week
Sponsored by: Dell EMC Isilon
The MANPATH environment variable behaviour is documented properly in the manpage
and it now has extended to new feature that allows to make MANPATH env variable
extending the default search path rather than overwriting it making the warning
painful
Reported by: kargl
MFC after: 1 week
mandoc.
If MANPATH begins with a colon, it is appended to the default list; if it ends
with a colon, it is prepended to the default list; or if it contains two
adjacent colons, the standard search path is inserted between the colons. If
none of these conditions are met, it overrides the standard search path.
Import the MANPATH description from mandoc into the man(1) man page
Reported by: kargl
MFC after: 1 week
localbase is not consistent with base for manpages:
/usr/local/man vs /usr/share/man adding share/man allows to fix that
inconsistency and would permit to remove tons of patches/modifications in the
ports tree
Some of the modifications from the previous summer of code has been integrated
Modification for compatibility with GNU diff output has been added
Main difference with OpenBSD:
Implement multiple GNU diff options:
* --ignore-file-name-case
* --no-ignore-file-name-case
* --normal
* --tabsize
* --strip-trailing-cr
Make diff -p compatible with GNU diff
Implement diff -l
Make diff -r compatible with GNU diff
Capsicumize diffing 2 regular files
Add a simple test suite
Approved by: AsiaBSDcon devsummit
Obtained from: OpenBSD, GSoC
Relnotes: yes
The localdef(1) changes are breaking world:
00:18:40.750 /usr/src/share/colldef/af_ZA.UTF-8.src: 2421: error: Bad file
descriptor
I will fix them offline.
Reported by: lwshu and many others
Also some small cleanups to match better current illumos.
CID: 1338540, 1338541, 1338557, 1338566
Obtained from: illumos
Discussed with: Yuri Pankov (@Nexenta)
MFC after: 5 days
As discussed during AsiaBSDcon devsummit, import the manpage from OpenBSD which
is has been rewritten in mdoc(7) format making it readable by default with
mandoc, it also has been extended by OpenBSD to cover all awk(1) options
Obtained from: OpenBSD
MFH: 1 week
- Use OBJTOP/SRCTOP-relative paths when looking for include files and
strfile.
- Add FORTUNES_OBJ and FORTUNES_SRC to abbreviate usr.bin/fortune
pathing.
This is being done to simplify make output/idioms.
MFC after: 1 week
Reviewed by: bdrewery
Sponsored by: Dell EMC Isilon
Differential Revision: D9916
In most cases strfile is built as part of build-tools, but in the event that someone
cd'ed to the directory, tried to build from scratch, and had MK_GAMES=no previously,
the build would fail in .../datfiles , trying to find strfile .
Mark this directory tree "SUBDIR_PARALLEL" safe to help facilitate this, instead of
shuffling around the SUBDIR entries (all of the other Makefiles will build standalone).
MFC after: 1 week
Sponsored by: Dell EMC Isilon
reallocarray(3) is a non portable extension from OpenBSD. Given that it is
already in FreeBSD, make easier future merges by adopting in some cases
where the code has some shared heritage with OpenBSD.
Obtained from: OpenBSD
by usually being double-spaced due to auto-wrap at column 80.
r212771 increased width of the hostname field from 12 to 25. This was
supposed to allow for 80-column output with all 3 load averages taking
5 characters each, but it actually gave width exactly 80 and thus worse
than useless auto-wrap in that case. 3 wide load average fields are
unusual, but later expansion of another field gave the auto-wrap with
just 2 wide load average fields.
Change to dynamic field widths for all fields except the uptime. This
also fixes the formatting of high (above 9999) user counts and not
very high (above 9.99) load averages. The formatting for numbers now
breaks at 99999.99, but scientific notation should be used starting
well below that.
The field width for the uptime remains hard-coded to work consistently
for uptimes less than 10000 days, but this gives too much space for
small uptimes. Punctuation between fields could be improved in many
ways, for example by removing it.
test suite
This change does the following:
- Introduces symmetry in the test inputs/outputs by adding the exit
code to the files. This simplified the test driver notably by
requiring less filename/test name manipulation.
- Adds a test driver for the testcases added in r313544, patterned
after bin/sh/tests/functional_test.sh . The driver calls indent as
noted in r313544, with an exception: The $FreeBSD$ RCS keyword's
expansion is reindented with indent, which means that the output
differs from the expected output. Thus, all lines with $FreeBSD$
in them are deleted on the fly, both in the input file and the
output file.
The test inputs/outputs are copied to the kyua sandbox before the
test is run as the pathing in some of the files relies on pathing
normalized to the current directory (copying the files is the
easiest way to resolve the issue).
Approved by: pstef (maintainer)
Reviewed by: pstef
X-MFC with: r313544
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D9682
For linux the mmap offset must also be page aligned, and we
need to disable macros like __FBSDID()
Change the linux osdep_uuidgen() to use more portable gettimeofday().
Reviewed by: marcel
Update libarchive to version 3.3.1 (and sync with latest vendor dist)
Notable vendor changes:
PR #501: improvements in ACL path handling
PR #724: fix hang when reading malformed cpio files
PR #864: fix out of bounds read with malformed GNU tar archives
Documentation, style, test suite improvements and typo fixes.
New options to bsdtar that enable or disable reading and/or writing of:
Access Control Lists (--acls, --no-acls)
Extended file flags (--fflags, --no-fflags)
Extended attributes (--xattrs, --no-xattrs)
Mac OS X metadata (Mac OS X only) (--mac-metadata, --no-mac-metadata)
MFC after: 2 weeks
Renumber cluase 4 to 3, per what everybody else did when BSD granted
them permission to remove clause 3. My insistance on keeping the same
numbering for legal reasons is too pedantic, so give up on that point.
Submitted by: Jan Schaumann <jschauma@stevens.edu>
Pull Request: https://github.com/freebsd/freebsd/pull/96
Prefer setvbuf() to setlinebuf() for portability.
Some style(9) and redundant tests for NULL.
These are only meant to ease up merging newer changes but we are skipping
changes done in order to accomodate OpenBSD's pledge support.
Obtained from: OpenBSD
MFC after: 2 weeks
r275234 addressed sort automatically converting 8-bit locales to UTF-8
by using "LANG=C sort", but LC_ALL overrides LANG if set, so the issue
may still be present depending on the user's environment. Use LC_ALL=C
instead.
Reported by: tests.reproducible-builds.org
Reviewed by: bapt
MFC after: 1 week
Sponsored by: The Linux Foundation / Core Infrastructure Initiative
Differential Revision: https://reviews.freebsd.org/D9765
The actual issue was the fact that if - was used then some restriction were
already set to stdin when we were applying caph_limit_stdio which was failing
due to the fact the fd was the fd was already restricted to lower rights.
Restricting stdio before actually opening the files prevent trying to raise the
right and fixes the issue.
And this allows to keep failing the program if restriction failed
Approved by: allanjude
Differential Revision: https://reviews.freebsd.org/D9723
When fed from a pipe, lam(1) would sometimes fail:
lam: unable to limit stdio: Capabilities insufficient
fixed regression in portsnap(8) introduced in r313938
This broke portsnap(8), the app that the capsicumization of lam(1) was
meant to secure.
# portsnap fetch update
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
Fetching snapshot tag from ec2-eu-west-1.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Tue Feb 21 16:05:39 MSK 2017 to Tue Feb 21 16:59:30 MSK 2017.
Fetching 5 metadata patches.lam: unable to limit stdio: Capabilities insufficient
done.
Applying metadata patches... done.
Fetching 5 metadata files... lam: unable to limit stdio: Capabilities insufficient
/usr/sbin/portsnap: cannot open 8c94d2c3f8fcea20eb1fd82021566c99c63a010e6b3702ee11e7a491795bcfb8.gz: No such file or directory
metadata is corrupt.
Reported by: Vladimir Zakharov <zakharov.vv@gmail.com>, Ben Woods <woodsb02@gmail.com>
MSDOS and Windows GNU grep uses -u to mean "print byte offsets as if
running on an UNIX system." The option has no effect on systems that
do not use CRLF line endings.
PR: 171200
Submitted by: deeptech71@gmail.com, Anders Jensen-Waud
MFC after: 1 month
Rework part of the loop in grep_fgetln to return the rest of the line
and ensure that we still advance the buffer by the length of the rest
of the line.
PR: 165471
Submitted by: Kyle Evans <kevans91@ksu.edu>
MFC after: 1 month
lam(1) is used in portsnap(8), so lock it down
Reviewed by: emaste, cem, jonathan
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D8076
* Initialize correct parent in binary_operator's constructor.
* Include <errno.h> explicitly, otherwise errno is undefined (without
NDEBUG, this is accidentally 'fixed' by including <iostream>).
Reported by: matteo
MFC after: 3 days
for maketab.c
The former simplifies pathing in make/displayed output, whereas the latter was just
unnecessarily superfluous since .PATH referenced the path to maketab.c earlier on in
the Makefile.
MFC after: 1 week
Sponsored by: Dell EMC Isilon
These examples show expected behavior of indent(1). They are meant to be used
together with a regression test mechanism, either Kyua, a Makefile or perhaps
something else. The mechanism should in essence do this:
indent -P${test}.pro < ${test}.0 > ${test}.0.run
and compare ${test}.0.stdout to ${test}.0.run. If the files differ or the exit
status isn't 0, the test failed.
* ${test}.pro is an indent(1) profile: a list of options passed through a file.
The program doesn't complain if the file doesn't exist.
* ${test}.0 is a C source file which acts as input for indent(1). It doesn't
have to have any particular formatting, since it's the output that matters.
* ${test}.0.stdout contains expected output. It doesn't have to be formatted in
Kernel Normal Form as the point of the tests is to check for regressions in
the program and not to check that it always produces KNF.
Reviewed by: ngie
Approved by: pfg (mentor)
Differential Revision: https://reviews.freebsd.org/D9007
In the kernel, cache the machine and flags fields from ELF header to use in
the ELF header of a core dump. For gcore, the copy these fields over from
the ELF header in the binary.
This matters for platforms which encode ABI information in the flags field
(such as o32 vs n32 on MIPS).
Reviewed by: kib
Sponsored by: DARPA / AFRL
Differential Revision: https://reviews.freebsd.org/D9392
The S_ISREG check was restored, such that the code will again fail with
in-place replacements on symlinks
MFC after: 12 days
X-MFC with: r313277
Sponsored by: Dell EMC Isilon
Small summary
-------------
o Almost all IPsec releated code was moved into sys/netipsec.
o New kernel modules added: ipsec.ko and tcpmd5.ko. New kernel
option IPSEC_SUPPORT added. It enables support for loading
and unloading of ipsec.ko and tcpmd5.ko kernel modules.
o IPSEC_NAT_T option was removed. Now NAT-T support is enabled by
default. The UDP_ENCAP_ESPINUDP_NON_IKE encapsulation type
support was removed. Added TCP/UDP checksum handling for
inbound packets that were decapsulated by transport mode SAs.
setkey(8) modified to show run-time NAT-T configuration of SA.
o New network pseudo interface if_ipsec(4) added. For now it is
build as part of ipsec.ko module (or with IPSEC kernel).
It implements IPsec virtual tunnels to create route-based VPNs.
o The network stack now invokes IPsec functions using special
methods. The only one header file <netipsec/ipsec_support.h>
should be included to declare all the needed things to work
with IPsec.
o All IPsec protocols handlers (ESP/AH/IPCOMP protosw) were removed.
Now these protocols are handled directly via IPsec methods.
o TCP_SIGNATURE support was reworked to be more close to RFC.
o PF_KEY SADB was reworked:
- now all security associations stored in the single SPI namespace,
and all SAs MUST have unique SPI.
- several hash tables added to speed up lookups in SADB.
- SADB now uses rmlock to protect access, and concurrent threads
can do SA lookups in the same time.
- many PF_KEY message handlers were reworked to reflect changes
in SADB.
- SADB_UPDATE message was extended to support new PF_KEY headers:
SADB_X_EXT_NEW_ADDRESS_SRC and SADB_X_EXT_NEW_ADDRESS_DST. They
can be used by IKE daemon to change SA addresses.
o ipsecrequest and secpolicy structures were cardinally changed to
avoid locking protection for ipsecrequest. Now we support
only limited number (4) of bundled SAs, but they are supported
for both INET and INET6.
o INPCB security policy cache was introduced. Each PCB now caches
used security policies to avoid SP lookup for each packet.
o For inbound security policies added the mode, when the kernel does
check for full history of applied IPsec transforms.
o References counting rules for security policies and security
associations were changed. The proper SA locking added into xform
code.
o xform code was also changed. Now it is possible to unregister xforms.
tdb_xxx structures were changed and renamed to reflect changes in
SADB/SPDB, and changed rules for locking and refcounting.
Reviewed by: gnn, wblock
Obtained from: Yandex LLC
Relnotes: yes
Sponsored by: Yandex LLC
Differential Revision: https://reviews.freebsd.org/D9352
The use of DES for anything is discouraged, especially with a static IV of 0
If you still need bdes(1) to decrypt Kirk's video lectures, see
security/bdes in ports.
This commit brought to you by the FOSDEM DevSummit and the
"remove unneeded dependancies on openssl in base" working group
Reviewed by: bapt, brnrd
Relnotes: yes
Sponsored by: FOSDEM DevSummit
Differential Revision: https://reviews.freebsd.org/D9424
Use S_ISREG instead of manual & (also it's better to compare the
result from & and the pattern instead of just assuming it's one bit
value).
Pointed out by Tianjie Mao <tjmao tjmao com>.
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D4827
Worker threads now use a pthread_cond_t to wait for work instead of
burning the cpu up.
Obtained from: DragonflyBSD (07774aea0ccf64a48fcfad8899e3bf7c8f18277a)
MFC after: 2 weeks
Note that mandoc does not use anymore sqlite3 but a home made database format
An important improvement has been made as well in makewhatis performance:
Tests on my laptop shows makewhatis on the entire system goes from 26s to 12s
result from & and the pattern instead of just assuming it's one bit
value).
Pointed out by Tianjie Mao <tjmao tjmao com>.
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D4827
Avoid "unused variable 'i'" warnings in generated .c files by only
emitting the "int i;" for non-opaque arrays. Opaque arrays use
xdr_opaque() rather than iterating over the array.
Obtained from: OpenBSD (CVS rev 1.28)
MFC after: 1 week
when fortune and other games moved from /usr/games to /usr/bin; I am
removing rather than correcting it since we normally do not mention in
the FILES section the paths to programs in /usr/bin/.
PR: 215962
Reported by: Andras Farkas
* Rewrite r_buf to use standard tail queues instead of a hand-rolled
circular linked list. Free dynamic allocations when done.
* Remove an optimization for the case where the file is a multiple of 128KB
in size and there is a scarcity of memory.
* Add ATF tests for "tail -r" and its variants.
Reported by: Valgrind
Reviewed by: ngie
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D9067
Instead of collecting statistics for each combination of ports and logical
units, that consumed ~45KB per LU with present number of ports, collect
separate statistics for every port and every logical unit separately, that
consume only 176 bytes per each single LU/port. This reduces struct
ctl_lun size down to just 6KB.
Also new IOCTL API/ABI does not hardcode number of LUs/ports, and should
allow handling of very large quantities.
MFC after: 2 weeks (probably keeping old API enabled for some time)
Load kvm symbols earlier to prevent rstat: symbol not in namelist
error when running netstat -rs.
Submitted by: Sebastian Huber <sebastian.huber@embedded-brains.de>
MFC after: 1 week
Sponsored by: Multiplay
invoked as llvm-ranlib, it can create an archive symbol table for
archives of objects compiled for LTO by an LLVM compiler.
Submitted by: Dan McGregor <danismostlikely@gmail.com>
MFC after: 3 days
Expand inet6name() line buffer to NI_MAXHOST and use strlcpy/snprintf
in various places.
Reported by: Anton Yuzhaninov <citrin citrin ru>
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D8916
It's been dead ever since it was imported from TI-RPC in 1995. The dead
code is still present in Illumos today, but was removed from NetBSD in 2006.
Reported by: Coverity
CID: 270097
Obtained from: NetBSD
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
It's always been dead, ever since first import in 1994. It's still dead in
OpenBSD's version, too.
Reported by: Coverity
CID: 270586
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
party software, this provides more standarized import workflow and
makes future upgrades easier.
The following files are new with this commit:
zconf.h.in
zlib.map
zlib.pc.in
They are not connected to build, but were kept in tree for reference
for future maintenance.
All our local trivial changes were applied to contrib/zlib, and the
contrib/zlib vendor source code is intended to 100% match lib/libz
before this commit.
MFC after: 2 weeks
As a followup to r310638, update libsysdecode (and kdump) to decode the
'mode' argument to getfsstat(). sysdecode_getfsstat_flags() has been
renamed to sysdecode_getfsstat_mode() and now treats the argument as an
enumerated value rather than a mask of flags.
Without this change, indent(1) would only look to load options from ~/.indent.pro if it's there and -npro wasn't used on the command line. This option lets the user set their own path to the file.
Approved by: pfg (mentor)
Differential Revision: https://reviews.freebsd.org/D9010
Move llvm-objdump from CLANG_EXTRAS to installed by default
We currently install three tools from binutils 2.17.50: as, ld, and
objdump. Work is underway to migrate to a permissively-licensed
tool-chain, with one goal being the retirement of binutils 2.17.50.
LLVM's llvm-objdump is intended to be compatible with GNU objdump
although it is currently missing some options and may have formatting
differences. Enable it by default for testing and further investigation.
It may later be changed to install as /usr/bin/objdump, it becomes a
fully viable replacement.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D8879
We currently install three tools from binutils 2.17.50: as, ld, and
objdump. Work is underway to migrate to a permissively-licensed
tool-chain, with one goal being the retirement of binutils 2.17.50.
LLVM's llvm-objdump is intended to be compatible with GNU objdump
although it is currently missing some options and may have formatting
differences. Enable it by default for testing and further investigation.
It may later be changed to install as /usr/bin/objdump, it becomes a
fully viable replacement.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D8879
zero-length array are dynamically sized at run-time based on the use
of hints, compilers can't be expected to figure out these offsets on
their own. [1]
- Fix incorrect comparison in cmp_nans(). [2]
PR: 204571 [1], 202301 [2]
Submitted by: David Binderman [2]
MFC after: 3 days
This takes the usual shortcut of only sandboxing the last input file.
It's a first cut and this program will be easy to adapt to sandbox all
files in the future.
iconv(1) has been changed to only open the conversion descriptor once,
since the input and output encodings are fixed over all inputs.
Instead, the descriptor is simply reset after each use (documented in
iconv(3) API).
Reviewed by: no one, unfortunately
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D7917
For now, only enter the sandbox for the last file processed (including
stdin for zero-argument mode).
Sandboxing all inputs will require a little restructuring of the
program.
Feedback by: emaste@ (earlier versions)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D7915
We restrict the (optional) input file and output files. It would be
nice to restrict the KVM files, but that's up to libkvm.
We wait until after kvm_nlist() is invoked to cap_enter() because
kldsym() isn't supported in the Capsicum sandbox.
Feedback from: emaste@ (earlier versions)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D7921
Separate dataset opening from reading/parsing. The number of input
files is already capped to a small number, so just open all input files
before sandboxing.
Feedback from: allanjude@ (earlier version), emaste@ (earlier version)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D7925
kinfo_proc::ki_tdname is three characters shorter than
thread::td_name. Add a ki_moretdname field for these three
extra characters. Add the new field to kinfo_proc32, as well.
Update all in-tree consumers to read the new field and assemble
the full name, except for lldb's HostThreadFreeBSD.cpp, which
I will handle separately. Bump __FreeBSD_version.
Reviewed by: kib
MFC after: 1 week
Relnotes: yes
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D8722
Avoid always using an O(n^2) loop over known syscall structures with
strcmp() on each system call. Instead, use a per-ABI cache indexed by
the system call number. The first 1024 system calls (which should cover
all of the normal system calls in currently-supported ABIs) use a flat array
indexed by the system call number to find system call structure. For other
system calls, a linked list of structures storing an integer to structure
mapping is stored in the ABI. The linked list isn't very smart, but it
should only be used by buggy applications invoking unknown system calls.
This also fixes handling of unknown system calls which currently trigger
a NULL pointer dereference.
Reviewed by: kib
MFC after: 2 weeks
ps.paren_indents:
When ps.paren_level was 0, this was accessing paren_indents[-1].
in_buffer:
This fragment checks if "*/" was read, but there's no guarantee that there
is more than one byte in the array (actually, this happens frequently for
the "{" in things like "int main(void) {").
Submitted by: Piotr Stefaniak
When special else-if processing is enabled (-ei), we can assume "else if"
and "if" to be equivalent for indentation purposes.
This reduction saves a lot of stack space in case of a long "if-else-if
... else-if" sequence; with this change,
Postgres/src/bin/psql/tab-complete.c as of 9.6beta3
requires minimum of the stack length to be 31 instead of 444.
Submitted by: Piotr Sephaniak
Teach indent(1) about storage-class specifiers. Don't assume
"in_parameter_declaration" state if "in_decl" hasn't been set. Don't set
"in_decl" for storage-class specifiers.
That set of changes helps with recognizing the difference between file
scope declarations like this:
static LIST_HEAD(, alq) ald_active;
static int ald_shuttingdown = 0;
struct thread *ald_thread;
and old style function declarators like this:
static int
do_execve(td, args, mac_p)
struct thread *td;
struct image_args *args;
struct mac *mac_p;
{
Unfortunately, at the same time this change makes indent(1) require
explicit int in declarations like "static a;", in order to understand that
it's part of a declaration. On the other hand, declarations like in the
first example are no longer indented as if ald_shuttingdown and ald_thread
were parameters of a function named LIST_HEAD.
Submitted by: Piotr Stefaniak
Do not set WARNS, so it gets the current default of 6.
Fix the warnings by sprinkling static, const, or strdup.
Make some constant data tables const. Fix whitespace.
MFC after: 1 week
Sponsored by: Dell EMC
Also, handle signed and unsigned chars, and more gracefully handle
invalid input.
Submitted by: bde in response to r309331
MFC after: 1 week
Sponsored by: Dell EMC
pr_comment() did avoid adding surplus space character when a comment
contained it at the end. Now it's also paying attention to tabs.
Taken from: Piotr Stefaniak
indent.c has a special loop that stores tokens from between an if () and
the next statement into a buffer. The loop ignored all newlines, but that
resulted in not calling dump_line() when it was needed to produce the
final line of the buffered up comment.
Taken from: Piotr Stefaniak
Work-around a somewhat complex interaction within the code. From
Piotr's commit [1]:
When pr_comment() calls dump_line() for the first line of a multiline
comment, it doesn't include any indentation - it starts with the "/*".
This is consistent for both boxed and not boxed comments. Where the logic
diverges is in how it treats the rest of the lines of the comment. For box
comments indent assumes that it must not change anything, so lines are
dumped as they were, including the indentation where it exists. For the
rest of comments, it will first remove the indentation to store plain text
of the comment and then add it again where indent thinks it's appropriate
-- this is part of comment re-indenting process.
For continuations of multi-line comments, the code that handles comments
in dump_line() will use pad_output() to create indentation from the
beginning of the line (what indent calls the first column) and then write
string pointed by s_com afterwards. But if it's a box comment, the string
will include original indentation, unless it's the first line of the
comment. This is why tab characters from s_com have to be considered when
calculating how much padding is needed and the "while (*com_st == '\t')
com_st++, target += 8;" does that.
In dump_line(), /target/ is initially set to ps.com_col, so it always
assumes that indentation needs to be produced in this function, regardless
of which line of a box comment it is. But for the first line of a box
comment it is not true, so pr_comment() signals it by setting
ps.n_comment_delta, the negative comment delta, to a negative number which
is then added to /target/ in dump_line() on all lines except the first
one, so that the function produces adequate indentation in this special
case.
The bug was in how that negative offset was calculated: pr_comment() used
count_spaces() on in_buffer, which pr_comment() expected to contain
non-null terminated sequence of characters, originating from whatever
originally was on the left side of the comment. Understanding that
count_spaces() requires a string, pr_comment() temporarily set buf_ptr[-2]
to 0 in hope that it would nul-terminate the right thing in in_buffer and
calling count_spaces() would be safe and do the expected thing. This was
false whenever buf_ptr would point into save_com, an entirely different
char array than in_buffer.
The short-term fix is to recognize whether buf_ptr points into in_buffer
or save_com.
Reference:
[1]
ea486a2aa3
Taken from: Piotr Stefaniak
This was needed on stable/10. Apparently, sys/param.h supplies CHAR_MAX
on head. Include limits.h anyway, for consistency, and because C says so.
Sponsored by: Dell EMC
when au_user_mask() fails, it's not a failure to set the audit mask,
but to calculate the audit mask -- and hence a condfiguration-file
issue (of some sort).
MFC after: 3 days
Sponsored by: DARPA, AFRL
During the upgrade of clang/llvm etc to 3.9.0 in r309124, the PACKAGE
directive in the usr.bin/clang/*.mk files got dropped accidentally.
Restore it, with a few minor changes and additions:
* Correct license in clang.ucl to NCSA
* Add PACKAGE=clang for clang and most of the "ll" tools
* Put lldb in its own package
* Put lld in its own package
Reviewed by: gjb, jmallett
X-MFC-With: 309124
Differential Revision: https://reviews.freebsd.org/D8666
indent(1) treated the "L" in "L'a'" as if it were an identifier and forced
a space character after it, breaking valid code.
PR: 143090
MFC after: 2 weeks
Multi-line comments are always block comments in KNF. Restore properly,
handling the case when a long one-liner gets wrapped and becomes a
multi-line comment.
Obtained from: Piotr Stefaniak
In C, strchr(3) returns a char*, whereas C++ defines two overloads:
* const char *strchr(const char*, int)
* char *strchr(char*, int)
Building fdt.cc (with the WITHOUT_GPL_DTC knob set) with libc++ 3.9.0 (imported
in r309124) was failing because libc++ r260377 added the first overload to
string.h, leading to failures such as:
fdt.cc:1638:8: error: cannot initialize a variable of type 'char *' with an
rvalue of type 'const char *'
Just define val as a const char* to fix it.
Upstreamed in https://github.com/davidchisnall/dtc/pull/14
Reviewed by: emaste
Approved by: emaste
If set it installs LLD as /usr/bin/ld. LLD (as of version 3.9) is not
capable of linking the world and kernel, but can self-host and link many
substantial applications. GNU ld continues to be used for the world and
kernel build, regardless of how this knob is set.
It is on by default for arm64, and off for all other CPU architectures.
Sponsored by: The FreeBSD Foundation
The "grouping" and "mon_grouping" values are arrays of one-byte
integers, not arrays of ASCII characters. Display them in a format
similar to GNU and MacOS.
MFC after: 3 days
Sponsored by: Dell EMC
sendfile_swapin() loop works this way:
- Find first invalid page in the request.
- Do vm_pager_has_page() and get count of pages, that can be taken in
single I/O.
- Trim valid pages from the end of the request.
- Cycle through the request and substitute to bogus_page all valid
pages that are in the middle of the request.
- After I/O launched (pager copies array of pages into buf(9), it
is important to restore proper page pointers with help vm_page_lookup().
Count bogus pages used and report them in sendfile stats.
An example problem case is 163.1.0.0 (University of Oxford)
which is in an APNIC ERX address range. Previously we assumed
that ARIN has the correct information for all ERX allocations,
but in this case ARIN refers back to APNIC, rather than referring
to RIPE. This caused whois to loop.
Whois will no longer loop back and forth forever between two RIRs
that don't have an answer, but instead try the other RIRs in turn.
As Jean-Sébastien notes, fold(1) requires handling argv-supplied files. That
will require a slightly more sophisticated approach.
Reported by: dumbbell@
Sponsored by: Dell EMC Isilon
Trivially capsicumize some simple programs that just interact with
stdio. This list of programs uses 'pledge("stdio")' in OpenBSD.
No objection from: allanjude, emaste, oshogbo
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8307
The previous calculation used an approximation which was only valid in
cases where the means being compared were similar; this resulted in very
odd claims being made, e.g. that 0 +/- 0 is a difference of -100% +/- 1%
from 100 +/- 1.
The new calculation scales sample standard deviations by the means, and
yields approximately correct percentage difference bounds providing that
the reference population is bounded away from zero. (In the case where
the values being compared are not sufficiently bounded away from zero,
the distribution of ratios becomes much harder to calculate, and is not
likely to be useful anyway.)
Note that when ministat is used for its intended purpose of determining
whether two samples are statistically different, this change is unlikely
to have any noticeable effect; in such cases the means will be similar
enough that the correction applied here will be minimal.
In r297602, which included a __FreeBSD_version bump to 1100105, we changed
sed 'i' and 'a' from discarding whitespaces to conform with what GNU and
sysvish sed do.
There are arguments in favor of keeping the old behavior but the new
behavior is also useful for migration purposes. It seems important to at
least consider the case of developers depending on the previous behavior,
so add a CFLAG to enable the old behaviour.
PR: 213474
MFC after: 5 days
The cornercase is when printenv is passed a parameter in the form VAR=val,
where VAR=val exists in the environment. In this case, printenv would print a
spurious newline and returns 0.
Approved by: cognet
MFC after: 1 week
those partitioning schemes that have this concept. Implement it as an
override for mbr's setting 0x80 in the flags for the first partition
when we have boot code.
Differential Revision: https://reviews.freebsd.org/D4403
UUIDs are not portable.
o Move mkimg_uuid() to a new file and merge both gpt_uuid_enc()
and vhd_uuid_enc() into a single mkimg_uuid_enc() that lives
in the same file.
o Move the OS-specific implementation of generating a UUID to
osdep_uuidgen() and provide the implementations for FreeBSD,
macOS and Linux.
o Expect the partitioning scheme headers to be found by having
a search to the directory in which the headers live. This
avoids conflicts on non-FreeBSD machines.
Restructure this script so that it generates a header of tables instead
of a source file. The tables are included in a flags.c source file which
provides functions to decode various system call arguments.
For functions that decode an enumeration, the function returns a pointer
to a string for known values and NULL for unknown values.
For functions that do more complex decoding (typically of a bitmask), the
function accepts a pointer to a FILE object (open_memstream() can be used
as a string builder) to which decoded values are written. If the
function operates on a bitmask, the function returns true if any bits
were decoded or false if the entire value was valid. Additionally, the
third argument accepts a pointer to a value to which any undecoded bits
are stored. This pointer can be NULL if the caller doesn't care about
remaining bits.
Convert kdump over to using decoder functions from libsysdecode instead of
mksubr. truss also uses decoders from libsysdecode instead of private
lookup tables, though lookup tables for objects not decoded by kdump remain
in truss for now. Eventually most of these tables should move into
libsysdecode as the automated table generation approach from mksubr is
less stale than the static tables in truss.
Some changes have been made to truss and kdump output:
- The flags passed to open() are now properly decoded in that one of
O_RDONLY, O_RDWR, O_WRONLY, or O_EXEC is always included in a decoded
mask.
- Optional arguments to open(), openat(), and fcntl() are only printed
in kdump if they exist (e.g. the mode is only printed for open() if
O_CREAT is set in the flags).
- Print argument to F_GETLK/SETLK/SETLKW in kdump as a pointer, not int.
- Include all procctl() commands.
- Correctly decode pipe2() flags in truss by not assuming full
open()-like flags with O_RDONLY, etc.
- Decode file flags passed to *chflags() as file flags (UF_* and SF_*)
rather than as a file mode.
- Fix decoding of quotactl() commands by splitting out the two command
components instead of assuming the raw command value matches the
primary command component.
In addition, truss and kdump now build without triggering any warnings.
All of the sysdecode manpages now include the required headers in the
synopsis.
Reviewed by: kib (several older versions), wblock (manpages)
MFC after: 2 months
Differential Revision: https://reviews.freebsd.org/D7847
r109332 introduced these three as DT_SUNW_*. Update to the correct
names already used elsewhere in FreeBSD and the Sun "Linker and
Libraries Guide"
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
- Numerous crash and bug fixes
- Improved warning and error messages
- Permit multiple labels on nodes and properties
- Fix node@address references
- Add support for /delete-node/
- Consume whitespace after a node
- Read the next token before the second /memreserve/
- Fix parsing of whitespace
- Clean up /delete-node/ and add support for /delete-property/
- Handle /delete-node/ specifying a unit address
Obtained from: https://github.com/davidchisnall/dtc @df5ede4
original commit log:
=====
I had originally suspected the parsing of ctype definition files as being
the source of the ctype flag mis-definitions, but it wasn't. In the
process, I simplified the cc_list parsing so I'm committing the no-impact
improvement separately. It removes some parsing redundancies and
won't parse partial range definitions anymore.
====
Submitted by: marino
Obtained from: Dragonfly
MFC after: 1 month
This commit is from John Marino in dragonfly with the following commit log:
====
This was a CTYPE encoding error involving consecutive points of the same
ctype. It was reported by myself to Illumos over a year ago but I was
unsure if it was only happening on BSD. Given the cause, the bug is also
present on Illumos.
Basically, if consecutive points were of the exact same ctype, they would
be defined as a range regardless. For example, all of these would be
considered equivalent:
<A> ... <C>, <H> (converts to <A> .. <H>)
<A>, <B>, <H> (converts to <A> .. <H>)
<A>, <J> ... <H> (converts to <A> .. <H>)
So all the points that shouldn't have been defined got "bridged" by the
extreme points.
The effects were recently reported to FreeBSD on PR 213013. There are
countless places were the ctype flags are misdefined, so this is a major
fix that has to be MFC'd.
====
This reveals a bad change I did on the testsuite: while 0x07FF is a valid
unicode it is not used yet (reserved for future use)
PR: 213013
Submitted by: marino@
Reported by: Kurtis Rader <krader@skepticism.us>
Obtained from: Dragonfly
MFC after: 1 month
Enter Capsicum capability sandbox pretty early in this setuid program.
Some minor modifications were needed to cache directory fds and use
relative lookups.
Rights restriction of the stdio descriptors is unfortunately pretty messy
because we need an ioctl capability not present in the current libcapsicum
helpers (FIODGNAME).
Reviewed by: ed
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D7999
In FreeBSD 11 ELF Tool Chain's elfcopy is installed as objcopy by
default, with the option to switch back to GNU objcopy by setting
WITHOUT_ELFCOPY_AS_OBJCOPY in make.conf.
We plan to remove the outdated in-tree binutils in FreeBSD 12, so
remove the temporary transition aid.
Reviewed by: brooks, imp
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D7337
inclusion of <sys/queue.h>.
Move the inclusion of the disk partitioning headers out of order
and inbetween standard headers and local header. They will change
in a subsequent commit.
be used on both macOS and Linux. STAILQs are not. In particular,
STAILQ_LAST does not next on Linux. Since neither STAILQ_FOREACH_SAFE
nor TAILQ_FOREACH_SAFE exist on Linux, replace its use with a regular
TAILQ_FOREACH. The _SAFE variant was only used for having the next
pointer in a local variable.
Previously, kdump used the kernel-only timervalsub() macro which required
defining _KERNEL when including <sys/time.h>. Now, kdump uses the existing
userland API. The timercmp() usage to check for a backwards timestamp is
also clearer and simpler than the previous code which checked the result of
the subtraction for a negative value.
While here, take advantage of the 3-arg timersub() to store the subtraction
results in a tempory timeval instead of overwriting the timestamp in the
ktrace record and then having to restore it.
The kernel uses a few negative errno values for internal conditions
such as requesting a system call restart. Normally these errno values
are not exposed to userland. However, kdump needs access to these
values as some of then can be present in a ktrace system call return
record. Previously kdump was defining _KERNEL to gain access to ehse
values, but was then having to manually declare 'errno' (and doing it
incorrectly). Now, kdump uses _WANT_KERNEL_ERRNO instead of _KERNEL
and uses the system-provided declaration of errno.
In particular, 64-bit system call arguments use up two register_t
arguments for 32-bit processes. They must also be aligned on a 64-bit
boundary on 32-bit powerpc processes. This fixes the decoding of
lseek(), procctl(), and wait6() arguments for 32-bit processes (both
native and via freebsd32).
Note that the ktrace system call return record only returns a single
register, so the return value of lseek is always truncated to the low
32-bits for 32-bit processes.
It also turns off dependencies (bsdinstall, bsdconfig, dpv, tzsetup).
Reviewed by: dteske
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D7969
1. macOS nor Linux have MAP_NOCORE nor MAP_NOSYNC. Define as 0.
2. macOS doesn't have SEEK_DATA nor SEEK_HOLE. Define as -1
so that lseek will return -1 (with errno set to EINVAL).
3. gcc correctly warns that error is assigned but not used in
image_copyout_region(). Fix by returning on the first error.
Not only is the header unportable, the encoding/decoding functions
are as well. Instead, duplicate the handful of small inlines we
need into a private header called endian.h.
Aside: an alternative approach is to move the encoding/decoding
functions to a separate system header. While the header is still
nonportable, such an approach would make it possible to re-use the
definitions by playing games with include paths. This may be the
preferred approach if more (build) utilities need this. This
change does not preclude that. In fact, it makes it easier.
otherwise format_resize(), which is called right after, isn't
getting the current/actual image size. Rather than rounding up,
format_resize() could end up truncating the size and we don't
allow that by design.
MFC after: 1 week
Closing stdin/stdout/stderr is often a bad idea as a future open()
can end up with its fd. Leave it open and limit it to no rights
instead.
Reviewed by: cem
Differential Revision: https://reviews.freebsd.org/D7984
bspatch previously included sys/capability.h or sys/capsicum.h based
on __FreeBSD_version, as FreeBSD is the upstream for bsdiff and we may
see this file incorporated into other third-party software.
The Capsicum header is now installed as sys/capsicum.h in stable/10 and
FreeBSD 10.3, so we can just use sys/capsicum.h and simplify the logic.
Reviewed by: allanjude
Differential Revision: https://reviews.freebsd.org/D7954
This is not actually documented or even implied in style(9). Make the change
to match convention. Someone should document this convention in style(9).
Reported by: jhb
Sponsored by: EMC Dell Isilon
This is a nice and trivial program for sandboxing. One input file, one
output file.
Reviewed by: pfg
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D7920
This is a straightforward single input, single output program for
capsicum.
Reviewed by: bapt
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D7928
stdio uses fstat and the TIOCGETA ioctl. Also collapse the
cap_rights_limit and new cap_ioctls_limit calls into one if statement.
Errors here are not actionable by the user and distinguishing stdout
from stderr doesn't really have value.
Reported by: kib
Reviewed by: allanjude, bapt
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D7944
if getcwd fails: just ignore it and do not try to adding to the list of possible
path where to find the files.
if fdopen fails, warn and return NULL the rest of the code knows how to deal
with it
Reported by: oshogbo
As a trick to be able to access all files passed in arguments (readonly) within
the sandbox we first open the root directory, then consider all files as
relative to this file descriptor.
This might be improved once casper add supports for filesystem.
MFC after: 1 month
Reviewed by: allanjude
Differential Revision: https://reviews.freebsd.org/D7936
errx() prefixes the error string with argv[0] so including "login: "
in the string is redundant. Also remove a superfluous newline.
Sponsored by: The FreeBSD Foundation
The first argument of calloc(3) should be an ordinal type, and the
second a size: split a multiplication to make better use of calloc(3)
and detect overflows.
Do some other re-ordering and style fixes while here.
MFC after: 3 weeks
Note that this introduces an explicit 2GB limit, but this was already
implicit in variable and function argument types.
This is based on the "non-cryptanalytic attacks against freebsd
update components" anonymous gist. Further refinement is planned.
Reviewed by: allanjude, cem, kib
Obtained from: anonymous gist
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D7619
process. We don't *quite* pull that number out of our backside, as
the actual number is difficult to determine without modifying the VM
system to report it, but it's still useful to get an idea of what's
going on when a machine unexpectedly starts swapping.
MFC after: 1 week
