* Add whitespace trimming to some fields (username, group, size, inode, blocks) to avoid whitespace in JSON strings
* fix -m mode, was invalid JSON (repeated keys), and was missing outer array container
* in -n mode, numeric uids and gids were returned as strings
Approved by: eadler (mentor)
Sponsored by: ScaleEngine Inc.
Differential Revision: https://reviews.freebsd.org/D2854
The message text is from cp, which has had a nicer message for this since
2007 (PR bin/50656).
As with cp, the exit status changes from 64 to 1.
PR: 201083
MFC after: 1 week
Fix shifts of possibly negative numbers found with ubsan and avoid signed
integer overflow when hashing an extremely long command name.
MFC after: 1 week
Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.
Still need to add real targets under targets/ to build packages.
Differential Revision: D2796
Reviewed by: brooks imp
using the color of symbolic link, the color is determined by the link target.
This behavior was quite confusing.
Looking at the file history, it looks like that r203665 intends to fix this
but the issue was never actually fixed.
Fix this by not setting FTS_COMFOLLOW when color is requested like what was
done in r203665.
MFC after: 2 weeks
This is also implemented in at least GNU coreutils cp.
While here also improve the '-l' handling to not open(2) the source file as
it does not actually need the descriptor.
Sponsored by: EMC / Isilon Storage Division
In this order, it won't try to recurse into a directory for which
it doesn't have permission, before changing that permission.
This follows an existing behavior in other BSDs, linux, OS X.
Obtained from: NetBSD
interpreted as a number, which checks the file's modification time and
use that as the date/time value.
This improves compatibility with GNU coreutils's version of time(1).
MFC after: 2 weeks
In an attempt to improve performance, cp reordered directories first
(although the comment says directories last). This is not effective with new
UFS layout policies.
The sorting reorders multiple arguments passed to cp, which may be
undesirable.
Additionally, the comparison function does not induce a total order. Per
POSIX, this causes undefined behaviour in qsort().
NetBSD removed the sorting in 2009.
On filesystems that return directory entries in hash/btree order, sorting by
d_fileno before statting improves performance on large directories. However,
this can only be implemented in fts(3).
PR: 53475
Reviewed by: bde (in 2004)
MFC after: 1 week
interactive.
I added the interactive check in r208881 to be safe, but in actual use
(scripts in set -m mode) passing along SIGINT seems best.
Discussed with: bdrewery
/usr/src/bin/ed/glbl.c:64:36: error: logical not is only applied to
theleft hand side of comparison [-Werror=logical-not-parentheses]
Obtained from: Dragonfly (1fff89cbaeaa43af720a1f23d9c466b756dd8a58)
MFC After: 1 month
The parser considered 'trap exit INT' to reset the default for both EXIT and
INT. This beahvior is not POSIX compliant. This was avoided if a value was
specified for 'exit', but then disallows exiting with the signal received. A
possible workaround is using ' exit'.
However POSIX does allow this type of behavior if the parameters are all
integers. Fix the handling for this and clarify its support in the manpage
since it is specifically allowed by POSIX.
Differential Revision: https://reviews.freebsd.org/D2325
Reviewed by: jilles
MFC after: 2 weeks
-O Force the archive to be one volume. If a volume ends prematurely, pax will
not prompt for a new volume.
PR: 198481
Submitted by: Sevan Janiyan
Reviewed by: allanjude (doc)
The previous 12h AM/PM format was perplexing as it didn't follow the
locale of the user and was a minor annoyance to FreeBSD users coming
from Linux. Additionally, the man page was incorrect about the strftime
format.
There are three time formats that may be displayed in the STARTED
column depending on the age of the process. Below is an example.
For a process started at 14:30 on Monday 16 March 2015, the following
formats may be used:
14:30 for process < 24h old (24h Timestamp)
Mon14 for process > 24h, < 1 week old (Weekday Hour)
16Mar15 for process > 1 week old (Day Month Year)
Differential Revision: https://reviews.freebsd.org/D1620
Reviewed by: brd
Approved by: trasz
r278742:
Simplify jail_name_to_jid and try to be more fault tolerant when scanning for
the jail ID (poll up to 10 times for the jail IDs to become available)
If the scan fails, the code will fall through and fail as it does with Jenkins
today
r278636:
Parameterize out the amount of sleep done in each test
Set the value in each test to a different amount to avoid potential
side-effects with other instances of the test (or lingering processes) still
being present on the system
r278633:
Refactor the tests
1. `id -u` -> 0 is now only checked once; the entire test script is now skipped
if this assertion is violated
2. De-dent whitespace, based on 1.
3. Only setup the symlink for $sleep once at the top of the script, and tear it
down once at the bottom of the script
the jail ID (poll up to 10 times for the jail IDs to become available)
If the scan fails, the code will fall through and fail as it does with Jenkins
today
Set the value in each test to a different amount to avoid potential
side-effects with other instances of the test (or lingering processes) still
being present on the system
1. `id -u` -> 0 is now only checked once; the entire test script is now skipped
if this assertion is violated
2. De-dent whitespace, based on 1.
3. Only setup the symlink for $sleep once at the top of the script, and tear it
down once at the bottom of the script
- Add special test to detect the case of -1 * INTMAX_MIN
- Protect against elimination of the test division by the optimizer
Garrett Cooper noticed that the overflow checks were incomplete, and Bruce
Evans suggested the use of the "volatile" qualifier to counter the effect
of the undefined behaviour, when the prior multiplication caused overflow,
and he also suggested improvements to the comments.
Reviewed by: bde
MFC after: 1 week
The core kernel part is patch file utimes.2008.4.diff from
pluknet@FreeBSD.org. I updated the code for API changes, added the manual
page and added compatibility code for old kernels. There is also audit and
Capsicum support.
A new UTIME_* constant might allow setting birthtimes in future.
Differential Revision: https://reviews.freebsd.org/D1426
Submitted by: pluknet (partially)
Reviewed by: delphij, pluknet, rwatson
Relnotes: yes
Some users build FreeBSD as non-root in Perforce workspaces. By default,
Perforce sets files read-only unless they're explicitly being edited.
As a result, the -f argument must be used to cp in order to override the
read-only flag when copying source files to object directories. Bare use of
'cp' should be avoided in the future.
Update all current users of 'cp' in the src tree.
Reviewed by: emaste
MFC after: 1 week
Sponsored by: Spectra Logic
EXP_REDIR was supposed to generate pathnames in redirection if exactly one
file matches, as permitted but not required by POSIX in interactive mode. It
is unlikely this will be implemented.
No functional change is intended.
MFC after: 1 week
Commands like 'export -p', 'set' and 'trap', and tracing enabled via 'set
-x' generate output suitable as shell input by adding quotes as necessary.
If there are control characters other than newline or invalid UTF-8
sequences, use $'...' and \OOO to display them safely.
The resulting output is not parsable by a strict POSIX.1-2008 shell but sh
from FreeBSD 9.0 and newer and many other shells can parse it.
feature is to quisce the system before suspend.
Stop is implemented by reusing the thread_single(9) with the special
mode SINGLE_ALLPROC. SINGLE_ALLPROC differs from the existing
single-threading modes by allowing (requiring) caller to operate on
other process. Interruptible sleeps for !TDF_SBDRY threads are
suspended like SIGSTOP does it, instead of aborting the sleep, like
SINGLE_NO_EXIT, to avoid spurious EINTRs on resume.
Provide debugging sysctl debug.stop_all_proc, which causes total stop
and suspends syncer, while waiting for variable reset for resume. It
is used for debugging; should be removed after the real use of the
interface is added.
In collaboration with: pho
Discussed with: avg
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
Per Austin Group issue #459, shifting zero positional parameters may or may
not be considered an operand error (which causes the shell to exit in most
cases).
EXP_REDIR was not being checked for while expanding positional parameters in
redirection, so CTL* bytes were not being prefixed where they should be.
MFC after: 1 week
dd(1) casts many of its numeric arguments from uintmax_t to intmax_t
and back again to detect whether or not the original arguments were
negative. This caused wrong behaviour in some boundary cases:
$ dd if=/dev/zero of=/dev/null count=18446744073709551615
dd: count cannot be negative
After the fix:
$ dd if=/dev/zero of=/dev/null count=18446744073709551615
dd: count: Result too large
PR: 191263
Submitted by: will@worrbase.com
Approved by: cognet@
POSIX does not permit to continuing a getopts loop with different
arguments. For parsing the positional parameters, we handle this case by
resetting the getopts state when the positional parameters are changed in
any way (and the getopts state is local to a function). However, in the
syntax getopts <optstring> <var> <arg...>, changes could lead to invalid
memory access.
In the syntax getopts <optstring> <var> <arg...>, store a copy of the
arguments and continue to use them until getopts is reset.
- use daemon(8) to write out a pid file for processes,
and check for for the existence of that file after
killing processes
- use explict named parameters to jail(8)
Makefiles should not assume that source files can be overwritten. This is the
common case for Perforce source trees.
This is a followup commit to r211243 in the same vein.
MFC after: 1 month
Sponsored by: Spectra Logic
MFSpectraBSD: r1036319 on 2014/01/29, r1046711 on 2014/03/06
Add a separate field which exports tracer pid and add a new keyword
("tracer") for ps to display it.
This is a follow up to r270444.
Reviewed by: kib
MFC after: 1 week
Relnotes: yes
1. 50+% of NO_PIE use is fixed by adding -fPIC to INTERNALLIB and other
build-only utility libraries.
2. Another 40% is fixed by generating _pic.a variants of various libraries.
3. Some of the NO_PIE use is a bit absurd as it is disabling PIE (and ASLR)
where it never would work anyhow, such as csu or loader. This suggests
there may be better ways of adding support to the tree. Many of these
cases can be fixed such that -fPIE will work but there is really no
reason to have it in those cases.
4. Some of the uses are working around hacks done to some Makefiles that are
really building libraries but have been using bsd.prog.mk because the code
is cleaner. Had they been using bsd.lib.mk then NO_PIE would not have
been needed.
We likely do want to enable PIE by default (opt-out) for non-tree consumers
(such as ports). For in-tree though we probably want to only enable PIE
(opt-in) for common attack targets such as remote service daemons and setuid
utilities. This is also a great performance compromise since ASLR is expected
to reduce performance. As such it does not make sense to enable it in all
utilities such as ls(1) that have little benefit to having it enabled.
Reported by: kib
These paths have had to be adjusted to changes in the testsuite runner
several times, so modify the tests to remove the need for such adjustment.
A cp in functional_test.sh is now unneeded, but this matters little in
performance.
In C, shift distances equal to or larger than the number of bits in the
operand result in undefined behaviour. As part of eliminating undefined
behaviour in arithmetic, mask off the distance like Java and JavaScript
specify and C on x86 usually does.
Assumption: conversion from unsigned to signed retains the two's complement
bits.
Assumption: uintmax_t has no padding bits.
Even though jail is part of the base system, it can be disabled by src.conf
settings. Therefore, it should be listed as a required program for tests
that use it.
CR: D603
MFC after: 3 days
Sponsored by: Spectra Logic
The new code uses a "test discovery mechanism" to determine
what tests are available for execution
The test shell can be specified via:
kyua test -v test_suites.FreeBSD.bin.sh.test_shell=/path/to/test/sh
Sponsored by: EMC / Isilon Storage Division
Approved by: jmmv (mentor)
Reviewed by: jilles (maintainer)
One problem is inferior(9) looping due to the process tree becoming a
graph instead of tree if the parent is traced by child. Another issue
is due to the use of p_oppid to restore the original parent/child
relationship, because real parent could already exited and its pid
reused (noted by mjg).
Add the function proc_realparent(9), which calculates the parent for
given process. It uses the flag P_TREE_FIRST_ORPHAN to detect the head
element of the p_orphan list and than stepping back to its container
to find the parent process. If the parent has already exited, the
init(8) is returned.
Move the P_ORPHAN and the new helper flag from the p_flag* to new
p_treeflag field of struct proc, which is protected by proctree lock
instead of proc lock, since the orphans relationship is managed under
the proctree_lock already.
The remaining uses of p_oppid in ptrace(PT_DETACH) and process
reapping are replaced by proc_realparent(9).
Phabric: D417
Reviewed by: jhb
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
variants. This allows usable file system images (i.e. those with both a
shell and an editor) to be created with only one copy of the curses library.
Exp-run: antoine
PR: 189842
Discussed with: bapt
Sponsored by: DARPA, AFRL
Currently, there can be no more than INT_MAX positional parameters. Make
sure to treat all higher ones as unset to avoid incorrect results and
crashes.
On 64-bit systems, our atoi() takes the low 32 bits of the strtol() and
sign-extends them.
On 32-bit systems, the call to atoi() returned INT_MAX for too high values
and there is not enough address space for so many positional parameters, so
there was no issue.
Although it is probably unwise to use this, POSIX is clear that leading
zeroes are permitted in positional parameters (and do not indicate octal).
Such positional parameters are checked for being unset and/or null
correctly, but their value is incorrectly expanded.
The test locale1.0 depends on locale support; it is meaningless without a
working LC_MESSAGES.
I added an OptionalObsoleteFiles.inc entry.
PR: 181151
Submitted by: Garrett Cooper (original version)
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
fflag to ignore fts_read errors, but stop deleting from that directory
because no further progress can be made.
When building a kernel with a high -j value on a high core count
machine, during the cleanobj phase we can wind up doing multiple rm
-rf at the same time for modules that have subdirectories. This
exposed this race (sometimes) as fts_read can return an error if the
directory is removed by another rm -rf. Since the intent of the -f
flag was to ignore errors, even if this was a bug in fts_read, we
should ignore the error like we've been instructed to do.
This is currently an opt-in build flag. Once ASLR support is ready and stable
it should changed to opt-out and be enabled by default along with ASLR.
Each application Makefile uses opt-out to ensure that ASLR will be enabled by
default in new directories when the system is compiled with PIE/ASLR. [2]
Mark known build failures as NO_PIE for now.
The only known runtime failure was rtld.
[1] http://www.bsdcan.org/2014/schedule/events/452.en.html
Submitted by: Shawn Webb <lattera@gmail.com>
Discussed between: des@ and Shawn Webb [2]
remove the now-redundant checks for RELEASE_CRUNCH. This originally
was defined for building smaller sysinstall images, but was later also
used by picobsd builds for a similar purpose. Now that we've moved
away from sysinstall, picobsd is the only remaining consumer of this
interface. Adding these two options reduces the RELEASE_CRUNCH
special cases in the tree by half.
When getopts finds an invalid option or a missing option-argument, it should
not reset its state and should set OPTIND as normal. This is an old ash bug
that was fixed long ago in dash. Our behaviour now matches most other
shells.
* Don't use sysexits.h. Just exit 1 on error and 0 otherwise.
* Don't sacrifice precision by converting the output of clock_gettime() to a
double and then comparing the results. Instead, subtract the values of
the two clock_gettime() calls, then convert to double.
* Don't use CLOCK_MONOTONIC_PRECISE. It's an unportable synonym for
CLOCK_MONOTONIC.
* Use more appropriate names for some local variables.
* In the summary message, round elapsed time to the nearest microsecond.
Reported by: bde, jilles
MFC after: 3 days
X-MFC-With: 265472
conv=sparse.
This change fixes two separate issues observed when the last output
block is all zeroes, and conv=sparse is in use. In this case, care
must be taken to roll back the last seek and write the entire last zero
block at the original offset where it should have occurred: when the
destination file is a block device, it is not possible to roll back
by just one character as the write would then not be properly aligned.
Furthermore, the buffer used to write this last all-zeroes block
needs to be properly zeroed-out. This was not the case previously,
resulting in a junk data byte appearing instead of a zero in the
output stream.
PR: bin/189174
PR: bin/189284
Reviewed by: kib
MFC after: 2 weeks
gettimeofday returns the system clock, which may jump forward or back,
especially if NTP is in use. If the time jumps backwards, then dd will see
negative elapsed time, round it up to 1usec, and print an absurdly fast
transfer rate.
The solution is to use clock_gettime(2) with CLOCK_MONOTONIC_PRECISE as the
clock_id. That clock advances steadily, regardless of changes to the system
clock.
Reviewed by: delphij
MFC after: 3 days
Sponsored by: Spectra Logic
As stated in the man page, this is equivalent to use "%a, %d %b %Y %T %z"
as the output format while LC_TIME is set to the "C" locale.
This is compatible with date(1) from the GNU core utilities.
From the original OpenBSD commit message:
restore the traditional behavior of -f implying -a; apparently Keith
Bostic forgot to restore it when the -f flag was put back on 2nd of
September 1989, after being removed on 16th of August as a
consequence of issues getting it working over NFS, so deviation from
traditional UNIX behavior in all BSDs looks like an historical
accident; as a side effect, this change accommodates behavior of
this option to IEEE Std 1003.1-2008 (``POSIX.1'').
joint work with jmc@ (who found the inaccuracy in our
implementation), schwarze@ (who provided a detailed tracking of
historical facts) and millert@
Submitted by: Igor Sobrado
Discussed with: mckusick
Obtained from: OpenBSD project
MFC after: 2 weeks
borrowed where syntax status=noxfer means no transfer statistics
and status=none means no status information at all.
This feature is useful because the statistics information can
sometimes be annoying, and redirecting stderr to /dev/null would
mean error messages also gets silenced.
Obtained from: OpenBSD
MFC after: 2 weeks
all the SUBDIR entries in parallel, instead of serially. Apply this
option to a selected number of Makefiles, which can greatly speed up the
build on multi-core machines, when using make -j.
This can be extended to more Makefiles later on, whenever they are
verified to work correctly with parallel building.
I tested this on a 24-core machine, with make -j48 buildworld (N = 6):
before stddev after stddev
======= ====== ======= ======
real time 1741.1 16.5 959.8 2.7
user time 12468.7 16.4 14393.0 16.8
sys time 1825.0 54.8 2110.6 22.8
(user+sys)/real 8.2 17.1
E.g. the build was approximately 45% faster in real time. On machines
with less cores, or with lower -j settings, the speedup will not be as
impressive. But at least you can now almost max out a machine with
buildworld!
Submitted by: jilles
MFC after: 2 weeks
Only store exit status for a process if that process has not terminated yet.
Test (slow):
exit 7 & p1=$!; until exit 8 & p2=$!; [ "$p1" = "$p2" ]; do wait "$p2";
done; sleep 0.1; wait %1; echo $?
should write "7".
Interestingly, the pkill tool lives in bin, not usr.bin. Haven't bothered
to check if this is because the tool moved or because the tests were
originally added in the wrong place.
When killing a %job started without job control, kill all processes in it.
As with process groups and zombies, if any process in the job can be killed
or has already terminated, the command is successful.
This also fixes occasional failures of the builtins/kill1.0 test.
Change {atf,plain,tap}.test.mk to be internal implementation details of
bsd.test.mk. Makefiles that build tests should now only include bsd.test.mk
and declaratively specify what they want to build, without worrying about
the internal implementation of the mk files.
The reason for this change is to permit building test programs of different
interfaces from a single directory, which is something I had a need for
while porting tests over from src/tools/regression/.
Additionally, this change makes it possible to perform some other requested
changes to bsd.test.mk in an easier manner. Coming soon.
If a job has terminated but is still known, silently do nothing when using
the kill builtin with the job specifier. Formerly, the shell called kill()
with the process group ID that might have been reused.
Redo expari() like evalvar(). This makes the logic more understandable and
avoids possible problems if arithmetic expansion occurs if CTLESC characters
are not generated (looking backwards for CTLARI is not generally possible in
that case but the old code tried anyway).
This adds an extra argstr() recursion.
If an alias's value ends with a space or tab, the next word is also
checked for aliases.
This is a POSIX feature. It is useful with utilities like command and
nohup (alias them to themselves followed by a space).
Add the space to avoid alias recursion when the alias is expanded, not when
it is added.
As a result, displaying an alias via command -v, command -V or type no
longer erroneously appends a space. Adjust the tests so they now require
this bug to be absent.
The SIGWINCH handler triggers breakage in libedit which is hard to fix; see
PR bin/169773.
Also, window size changes while a program is in foreground (and it rather
than sh will receive SIGWINCH) will now be picked up automatically.
Downside: it is now certain that a resize is only processed after pressing
<Enter>. If libedit is fixed, sh will most likely have to be changed also.
PR: bin/180146
Redo r260506 by using the new TEST_METADATA functionality of bsd.test.mk
to mark the sh(1) and test(1) tests as not supporting root. This is to
get rid of hand-crafted Kyuafiles for these very simple cases.
MFC after: 5 days
Redo r260586 by using the new functionality in tap.test.mk to transparently
support perl-based test programs.
As a side-effect, we get rid of an explicit path to /usr/bin/perl by
replacing it with /usr/local/bin/perl (or as defined in tap.test.mk).
This also fixes the name of the legacy_test source file because this should
have always been legacy_test.pl and not legacy_test.sh. My mistake when
originally moving the code around without realizing that this was a perl
script.
MFC after: 5 days
The effect of this is that the test program is marked as skipped when perl
is missing, instead of marking it as broken due to an execution failure.
MFC after: 3 days
One of the tests for test(1) fails and some of the tests for sh(1) are
silently bypassed when running as root.
To fix these tests and ensure they all run, mark the test programs for
sh(1) and test(1) as requiring an unprivileged user. (This should and
will be the default in Kyua but isn't yet.)
MFC after: 1 week
preadbuffer() maintained a flag whether there was any non-whitespace
character. This flag is only useful when history is enabled (in that case,
lines containing only whitespace are not added to history). Instead, check
using strspn() when history is enabled.
There is an approximate 2% speedup when running
sh -c '. /etc/rc.subr; . /etc/defaults/rc.conf; source_rc_confs'
with hot cache.
