The first file in these lists will generate everything else so only
it should be getting a .meta file. With bmake's missing=yes meta
feature these would otherwise cause a rebuild without the
.NOMETA hint.
Sponsored by: EMC / Isilon Storage Division
case of \c in the prompt format string is a no-op. We already passed
this test at the top of the loop, and i has not yet been incremented in
this path. Change this test to (i < PROMPTLEN - 2).
Reported by: Coverity
CID: 1008328
Reviewed by: cem
MFC after: 1 week
Replace the magic constant 127 in the loop interation count with
"PROMPTLEN - 1".
gethostname() is not guaranteed to NUL terminate the destination
string if it is too short. Decrease the length passed to gethostname()
by one, and add a NUL at the end of the buffer to make sure the
following loop to find the end of the name properly terminates.
The default: case is the likely cause of Coverity CID 1008328. If
i is 126 at the top of the loop interation where the default case
is triggered, i will be incremented to 127 by the default case,
then incremented to 128 at the top of the loop before being compared
to 127 (PROMPTLENT - 1) and terminating the loop. Then the NUL
termination code after the loop will write to ps[128]. Fix by
checking for overflow before incrementing the index and storing the
second character in the buffer.
These fixes are not guaranteed to satisfy Coverity. The code that
increments i in the 'h'/'H' and 'w'/'W' cases may be beyond its
capability to analyze, but the code appears to be safe.
Reported by: Coverity
CID: 1008328
Reviewed by: jilles, cem
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D6482
after r298107
Summary of changes:
- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)
Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.
MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division
If the hostname is empty and \h is used in $PS1,
the remainder of the prompt following \h will be empty.
Likewise for $PWD and \w. Fix it.
Reviewed by: jilles
MFC after: 1 week
Sponsored by: Dell Inc.
Differential Revision: https://reviews.freebsd.org/D6188
This avoids 'build command changed' due to CFLAGS/CC changes during the
normal build. Without this the build-tools targets end up rebuilding
for the *target* rather than keeping the native versions built in
build-tools.
Sponsored by: EMC / Isilon Storage Division
breaking the ABI. Special value is stored in the lock pointer to
indicate shared lock, and offline page in the shared memory is
allocated to store the actual lock.
Reviewed by: vangyzen (previous version)
Discussed with: deischen, emaste, jhb, rwatson,
Martin Simmons <martin@lispworks.com>
Tested by: pho
Sponsored by: The FreeBSD Foundation
These are no longer needed after the recent 'beforebuild: depend' changes
and hooking DIRDEPS_BUILD into a subset of FAST_DEPEND which supports
skipping 'make depend'.
Sponsored by: EMC / Isilon Storage Division
Note: tcsh(1) has a MK_TCSH=no test, so this should be a separate
package, which requires pre-install/post-install scripts, to be
added later.
Sponsored by: The FreeBSD Foundation
Builtins (including variable assignments without command word), function
calls and redirected compound commands need to restore file descriptors
to their original state after execution. This is handled by allocating a
redirtab structure. These mallocs and frees show up heavily in pmcstat.
Only allocate a redirtab if there are actually redirections and maintain a
count of how many levels of REDIR_PUSH there are without redirtabs.
A simple loop without external programs like
sh -c 'i=0; w=$(printf %0100d 7); while [ "$i" -lt 1000000 ]; do
i=$((i+1)); done'
is over 25% faster on an amd64 bhyve VM.
Forbid (temporary or permanent) modifications of the strings in NARG nodes
during expansion.
Tilde expansion now needs to copy the username for the terminating '\0'.
Per POSIX, 'cd -' should use the OLDPWD shell variable, not internal state.
This variable is normally exported.
Also, if OLDPWD is not set, fail 'cd -' instead of changing to the current
directory.
Some variables like PATH call a function when modified. Make sure to call
this also when leaving a function where such a variable was made local.
Make sure to restore local variables before shellparam, so getopts state is
not clobbered.
Redirect 'cd -' output to /dev/null since POSIX requires it to write the new
directory name even if not interactive, but we currently only write it if
interactive.
Command substitutions containing a single simple command and here-document
expansion are performed in a subshell environment, but may not fork. Any
modified state of the shell environment should be restored afterward.
The state that OPTIND=1 had been done was not saved and restored here.
Note that the other parts of shellparam need not be saved and restored,
since they are not modified in these situations (a fork is done before such
modifications).
With the new expand.c code, the intermediate representation passed to the
pathname generation code only contains CTLESC, not CTLQUOTEMARK.
CTLQUOTEMARK now only occurs in the text of NARG nodes (output of the
parser).
This avoids the need to add and remove CTLESC bytes if pathname generation
will not be performed (set -f).
Side effect: the order of operations is slightly different: pathname
generation in ${$+* $(CMD)} will not see filesystem changes from CMD.
This simplifies the code and should be faster in some cases.
Side effect: the order of operations is different so that the value of IFS
used when IFS is modified during expansion (${IFS:=...}, ${IFS=...} or
$((...IFS=...))) may be different. Note that this order is highly unportable
between shells.
netbsd-tests.test.mk (r289151)
- Eliminate explicit OBJTOP/SRCTOP setting
- Convert all ad hoc NetBSD test integration over to netbsd-tests.test.mk
- Remove unnecessary TESTSDIR setting
- Use SRCTOP where possible for clarity
MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Divison
This simplifies the code (e.g. allowing use of qsort(3) instead of a
hand-rolled mergesort) and should have better cache properties.
The waste of unused args arrays after resizes is approximately the same as
the savings from getting rid of the next pointers.
At the same time, remove a piece of global state and move some duplicated
code into a function.
Shell syntax is too complicated to detect command substitution and unquoted
operators reliably without implementing much of sh's parser. Therefore, have
sh do this detection.
While changing sh's support anyway, also read input from a pipe instead of
arguments to avoid {ARG_MAX} limits and improve privacy, and output count
and length using 16 instead of 8 digits.
The basic concept is:
execl("/bin/sh", "sh", "-c", "freebsd_wordexp ${1:+\"$1\"} -f "$2",
"", flags & WRDE_NOCMD ? "-p" : "", <pipe with words>);
The WRDE_BADCHAR error is still implemented in libc. POSIX requires us to
fail strings containing unquoted braces with code WRDE_BADCHAR. Since this
is normally not a syntax error in sh, there is still a need for checking
code in libc, we_check().
The new we_check() is an optimistic check that all the characters
<newline> | & ; < > ( ) { }
are quoted. To avoid duplicating too much sh logic, such characters are
permitted when quoting characters are seen, even if the quoting characters
may themselves be quoted. This code reports all WRDE_BADCHAR errors; bad
characters that get past it and are a syntax error in sh return WRDE_SYNTAX.
Although many implementations of WRDE_NOCMD erroneously allow some command
substitutions (and ours even documented this), there appears to be code that
relies on its security (codesearch.debian.net shows quite a few uses).
Passing untrusted data to wordexp() still exposes a denial of service
possibility and a fairly large attack surface.
Reviewed by: wblock (man page only)
MFC after: 2 weeks
Relnotes: yes
Security: fixes command execution with wordexp(untrusted, WRDE_NOCMD)
POSIX requires this to prevent entering function definitions in history but
this implementation does nothing except retain the option's value. In ksh88,
function definitions were usually entered in the history file, even when
they came from ~/.profile and the $ENV file, to allow displaying their
definitions.
This is also the first option that does not have a letter.
The initial check for a matching ] was incorrect if a ] may be consumed by a
[:class:]. The subsequent loop assumed that there must be a ].
Remove the initial check and make the loop cope with a missing ].
Found with afl-fuzz.
MFC after: 1 week
An invalid substitution like ${var@} does not cause a parse error but is
stored in the intermediate representation, to be written as part of the
error message. If there is a CTL* byte in the stored part, this confuses
some code such as the code to skip an unused alternative such as in
${var-alternative}.
To keep things simple, do not store CTL* bytes.
Found with afl-fuzz.
MFC after: 1 week
The negative value was not expected and generated the low 8 bits as a byte,
which may be an invalid character encoding.
The final shift in creating the negative value was undefined as well.
Make the temporary variable unsigned to fix this.
Fix shifts of possibly negative numbers found with ubsan and avoid signed
integer overflow when hashing an extremely long command name.
MFC after: 1 week
Off by default, build behaves normally.
WITH_META_MODE we get auto objdir creation, the ability to
start build from anywhere in the tree.
Still need to add real targets under targets/ to build packages.
Differential Revision: D2796
Reviewed by: brooks imp
interactive.
I added the interactive check in r208881 to be safe, but in actual use
(scripts in set -m mode) passing along SIGINT seems best.
Discussed with: bdrewery
The parser considered 'trap exit INT' to reset the default for both EXIT and
INT. This beahvior is not POSIX compliant. This was avoided if a value was
specified for 'exit', but then disallows exiting with the signal received. A
possible workaround is using ' exit'.
However POSIX does allow this type of behavior if the parameters are all
integers. Fix the handling for this and clarify its support in the manpage
since it is specifically allowed by POSIX.
Differential Revision: https://reviews.freebsd.org/D2325
Reviewed by: jilles
MFC after: 2 weeks
EXP_REDIR was supposed to generate pathnames in redirection if exactly one
file matches, as permitted but not required by POSIX in interactive mode. It
is unlikely this will be implemented.
No functional change is intended.
MFC after: 1 week
Commands like 'export -p', 'set' and 'trap', and tracing enabled via 'set
-x' generate output suitable as shell input by adding quotes as necessary.
If there are control characters other than newline or invalid UTF-8
sequences, use $'...' and \OOO to display them safely.
The resulting output is not parsable by a strict POSIX.1-2008 shell but sh
from FreeBSD 9.0 and newer and many other shells can parse it.
Per Austin Group issue #459, shifting zero positional parameters may or may
not be considered an operand error (which causes the shell to exit in most
cases).
EXP_REDIR was not being checked for while expanding positional parameters in
redirection, so CTL* bytes were not being prefixed where they should be.
MFC after: 1 week
POSIX does not permit to continuing a getopts loop with different
arguments. For parsing the positional parameters, we handle this case by
resetting the getopts state when the positional parameters are changed in
any way (and the getopts state is local to a function). However, in the
syntax getopts <optstring> <var> <arg...>, changes could lead to invalid
memory access.
In the syntax getopts <optstring> <var> <arg...>, store a copy of the
arguments and continue to use them until getopts is reset.
These paths have had to be adjusted to changes in the testsuite runner
several times, so modify the tests to remove the need for such adjustment.
A cp in functional_test.sh is now unneeded, but this matters little in
performance.
In C, shift distances equal to or larger than the number of bits in the
operand result in undefined behaviour. As part of eliminating undefined
behaviour in arithmetic, mask off the distance like Java and JavaScript
specify and C on x86 usually does.
Assumption: conversion from unsigned to signed retains the two's complement
bits.
Assumption: uintmax_t has no padding bits.
The new code uses a "test discovery mechanism" to determine
what tests are available for execution
The test shell can be specified via:
kyua test -v test_suites.FreeBSD.bin.sh.test_shell=/path/to/test/sh
Sponsored by: EMC / Isilon Storage Division
Approved by: jmmv (mentor)
Reviewed by: jilles (maintainer)
variants. This allows usable file system images (i.e. those with both a
shell and an editor) to be created with only one copy of the curses library.
Exp-run: antoine
PR: 189842
Discussed with: bapt
Sponsored by: DARPA, AFRL
Currently, there can be no more than INT_MAX positional parameters. Make
sure to treat all higher ones as unset to avoid incorrect results and
crashes.
On 64-bit systems, our atoi() takes the low 32 bits of the strtol() and
sign-extends them.
On 32-bit systems, the call to atoi() returned INT_MAX for too high values
and there is not enough address space for so many positional parameters, so
there was no issue.
Although it is probably unwise to use this, POSIX is clear that leading
zeroes are permitted in positional parameters (and do not indicate octal).
Such positional parameters are checked for being unset and/or null
correctly, but their value is incorrectly expanded.
The test locale1.0 depends on locale support; it is meaningless without a
working LC_MESSAGES.
I added an OptionalObsoleteFiles.inc entry.
PR: 181151
Submitted by: Garrett Cooper (original version)
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
When getopts finds an invalid option or a missing option-argument, it should
not reset its state and should set OPTIND as normal. This is an old ash bug
that was fixed long ago in dash. Our behaviour now matches most other
shells.
Only store exit status for a process if that process has not terminated yet.
Test (slow):
exit 7 & p1=$!; until exit 8 & p2=$!; [ "$p1" = "$p2" ]; do wait "$p2";
done; sleep 0.1; wait %1; echo $?
should write "7".
When killing a %job started without job control, kill all processes in it.
As with process groups and zombies, if any process in the job can be killed
or has already terminated, the command is successful.
This also fixes occasional failures of the builtins/kill1.0 test.
Change {atf,plain,tap}.test.mk to be internal implementation details of
bsd.test.mk. Makefiles that build tests should now only include bsd.test.mk
and declaratively specify what they want to build, without worrying about
the internal implementation of the mk files.
The reason for this change is to permit building test programs of different
interfaces from a single directory, which is something I had a need for
while porting tests over from src/tools/regression/.
Additionally, this change makes it possible to perform some other requested
changes to bsd.test.mk in an easier manner. Coming soon.
If a job has terminated but is still known, silently do nothing when using
the kill builtin with the job specifier. Formerly, the shell called kill()
with the process group ID that might have been reused.
Redo expari() like evalvar(). This makes the logic more understandable and
avoids possible problems if arithmetic expansion occurs if CTLESC characters
are not generated (looking backwards for CTLARI is not generally possible in
that case but the old code tried anyway).
This adds an extra argstr() recursion.
If an alias's value ends with a space or tab, the next word is also
checked for aliases.
This is a POSIX feature. It is useful with utilities like command and
nohup (alias them to themselves followed by a space).
Add the space to avoid alias recursion when the alias is expanded, not when
it is added.
As a result, displaying an alias via command -v, command -V or type no
longer erroneously appends a space. Adjust the tests so they now require
this bug to be absent.
The SIGWINCH handler triggers breakage in libedit which is hard to fix; see
PR bin/169773.
Also, window size changes while a program is in foreground (and it rather
than sh will receive SIGWINCH) will now be picked up automatically.
Downside: it is now certain that a resize is only processed after pressing
<Enter>. If libedit is fixed, sh will most likely have to be changed also.
PR: bin/180146
Redo r260506 by using the new TEST_METADATA functionality of bsd.test.mk
to mark the sh(1) and test(1) tests as not supporting root. This is to
get rid of hand-crafted Kyuafiles for these very simple cases.
MFC after: 5 days
One of the tests for test(1) fails and some of the tests for sh(1) are
silently bypassed when running as root.
To fix these tests and ensure they all run, mark the test programs for
sh(1) and test(1) as requiring an unprivileged user. (This should and
will be the default in Kyua but isn't yet.)
MFC after: 1 week
preadbuffer() maintained a flag whether there was any non-whitespace
character. This flag is only useful when history is enabled (in that case,
lines containing only whitespace are not added to history). Instead, check
using strspn() when history is enabled.
There is an approximate 2% speedup when running
sh -c '. /etc/rc.subr; . /etc/defaults/rc.conf; source_rc_confs'
with hot cache.
This change is a proof of concept on how to easily integrate existing
tests from the tools/regression/ hierarchy into the /usr/tests/ test
suite and on how to adapt them to the new layout for src.
To achieve these goals, this change:
- Moves tests from tools/regression/bin/<tool>/ to bin/<tool>/tests/.
- Renames the previous regress.sh files to legacy_test.sh.
- Adds Makefiles to build and install the tests and all their supporting
data files into /usr/tests/bin/.
- Plugs the legacy_test test programs into the test suite using the new
TAP backend for Kyua (appearing in 0.8) so that the code of the test
programs does not have to change.
- Registers the new directories in the BSD.test.dist mtree file.
Reviewed by: freebsd-testing
Approved by: rpaulo (mentor)
Although <&0 does nothing, it is a redirection affecting standard input and
should therefore disable the </dev/null redirection implicit in a background
command.
If job control is not enabled, background jobs started with ... & ignore
SIGINT and SIGQUIT so that they are not affected by such signals that are
intended for the foreground job. However, this should not prevent
reassigning a different action for these signals (as if the shell invocation
inherited these signal actions from its parent).
Austin group issue #751
Example:
{ trap - INT; exec sleep 10; } & wait
A Ctrl+C should terminate the sleep command.
user. Kqueue now saves the ucred of the allocating thread, to
correctly decrement the counter on close.
Under some specific and not real-world use scenario for kqueue, it is
possible for the kqueues to consume memory proportional to the square
of the number of the filedescriptors available to the process. Limit
allows administrator to prevent the abuse.
This is kernel-mode side of the change, with the user-mode enabling
commit following.
Reported and tested by: pho
Discussed with: jmg
Sponsored by: The FreeBSD Foundation
MFC after: 2 weeks
The getpgrp() call is unnecessary: if there is no job control then the
result was not used at all and if there is job control then we are not a
subshell and our process group ID is equal to our process ID (rootpid).
Formerly, return always returned from a function if it was called from a
function, even if there was a closer dot script. This was for compatibility
with the Bourne shell which only allowed returning from functions.
Other modern shells and POSIX return from the function or the dot script,
whichever is closest.
Git 1.8.4's rebase --continue depends on the POSIX behaviour.
Reported by: Christoph Mallon, avg
The change in r238888 was incomplete. It was still possible for a trapped
signal to arrive before the shell went to sleep (sigsuspend()) because a
check was missing or because the signal arrived before in_waitcmd was set.
On SMP, this bug sometimes caused the builtins/wait4.0 test to take 1 second
to execute; it then might or might not fail. On UP, the test almost always
failed.
The erflag argument was only used by old-style (``) command substitutions.
We can remove it and handle the special case in the command substitution
code.