Shell syntax is too complicated to detect command substitution and unquoted
operators reliably without implementing much of sh's parser. Therefore, have
sh do this detection.
While changing sh's support anyway, also read input from a pipe instead of
arguments to avoid {ARG_MAX} limits and improve privacy, and output count
and length using 16 instead of 8 digits.
The basic concept is:
execl("/bin/sh", "sh", "-c", "freebsd_wordexp ${1:+\"$1\"} -f "$2",
"", flags & WRDE_NOCMD ? "-p" : "", <pipe with words>);
The WRDE_BADCHAR error is still implemented in libc. POSIX requires us to
fail strings containing unquoted braces with code WRDE_BADCHAR. Since this
is normally not a syntax error in sh, there is still a need for checking
code in libc, we_check().
The new we_check() is an optimistic check that all the characters
<newline> | & ; < > ( ) { }
are quoted. To avoid duplicating too much sh logic, such characters are
permitted when quoting characters are seen, even if the quoting characters
may themselves be quoted. This code reports all WRDE_BADCHAR errors; bad
characters that get past it and are a syntax error in sh return WRDE_SYNTAX.
Although many implementations of WRDE_NOCMD erroneously allow some command
substitutions (and ours even documented this), there appears to be code that
relies on its security (codesearch.debian.net shows quite a few uses).
Passing untrusted data to wordexp() still exposes a denial of service
possibility and a fairly large attack surface.
Reviewed by: wblock (man page only)
MFC after: 2 weeks
Relnotes: yes
Security: fixes command execution with wordexp(untrusted, WRDE_NOCMD)
An invalid substitution like ${var@} does not cause a parse error but is
stored in the intermediate representation, to be written as part of the
error message. If there is a CTL* byte in the stored part, this confuses
some code such as the code to skip an unused alternative such as in
${var-alternative}.
To keep things simple, do not store CTL* bytes.
Found with afl-fuzz.
MFC after: 1 week
The negative value was not expected and generated the low 8 bits as a byte,
which may be an invalid character encoding.
The final shift in creating the negative value was undefined as well.
Make the temporary variable unsigned to fix this.
If an alias's value ends with a space or tab, the next word is also
checked for aliases.
This is a POSIX feature. It is useful with utilities like command and
nohup (alias them to themselves followed by a space).
The erflag argument was only used by old-style (``) command substitutions.
We can remove it and handle the special case in the command substitution
code.
NEOF needs to be a non-null pointer distinct from valid union node pointers.
It is not dereferenced.
The new NEOF is much like SIG_ERR except that it is an object pointer
instead of a function pointer.
The variable tokpushback can now be static.
As per POSIX, a simple command must have at least one redirection,
assignment word or command word.
These occured in rare cases such as eval "f()" .
The extension of allowing no commands inside { }, if, while, for, etc.
remains.
Replace the RESET blocks with regular functions and a reset() function that
calls them all.
This code generation tool is unusual and does not appear to provide much
benefit. I do not think isolating the knowledge about which modules need to
be reset is worth an almost 500-line build tool and wider scope for
variables used by the reset functions.
Also, relying on reset functions is often wrong: the cleanup should be done
in exception handlers so that no stale state remains after 'command eval'
and the like.
It now passes WARNS=7 with clang on i386.
GCC 4.2.1 does not understand setjmp() properly so will always trigger
-Wuninitialized. I will not add the volatile keywords to suppress this.
Replacing ;; with the new control operator ;& will cause the next list to be
executed as well without checking its pattern, continuing until a list ends
with ;; or until the end of the case statement. This is like omitting
"break" in a C "switch" statement.
The sequence ;& was formerly invalid.
This feature is proposed for the next POSIX issue in Austin Group issue
#449.
The function name expandstr() and the general idea of doing this kind of
expansion by treating the text as a here document without end marker is from
dash.
All variants of parameter expansion and arithmetic expansion also work (the
latter is not required by POSIX but it does not take extra code and many
other shells also allow it).
Command substitution is prevented because I think it causes too much code to
be re-entered (for example creating an unbounded recursion of trace lines).
Unfortunately, our LINENO is somewhat crude, otherwise PS4='$LINENO+ ' would
be quite useful.
This is sometimes used with eval or old-style command substitution, and most
shells other than ash derivatives allow it.
It can also be used with scripts that violate POSIX's requirement on the
application that they end in a newline (scripts must be text files except
that line length is unlimited).
Example:
v=`cat <<EOF
foo
EOF`
echo $v
This commit does not add support for the similar construct with new-style
command substitution, like
v=$(cat <<EOF
foo
EOF)
This continues to require a newline after the terminator.
Because we have no iconv in base, support for other charsets is not
possible.
Note that \u/\U are processed using the locale that was active when the
shell started. This is necessary to avoid behaviour that depends on the
parse/execute split (for example when placing braces around an entire
script). Therefore, UTF-8 encoding is implemented manually.
A string between $' and ' may contain backslash escape sequences similar to
the ones in a C string constant (except that a single-quote must be escaped
and a double-quote need not be). Details are in the sh(1) man page.
This construct is useful to include unprintable characters, tabs and
newlines in strings; while this can be done with a command substitution
containing a printf command, that needs ugly workarounds if the result is to
end with a newline as command substitution removes all trailing newlines.
The construct may also be useful in future to describe unprintable
characters without needing to write those characters themselves in 'set -x',
'export -p' and the like.
The implementation attempts to comply to the proposal for the next issue of
the POSIX specification. Because this construct is not in POSIX.1-2008,
using it in scripts intended to be portable is unwise.
Matching the minimal locale support in the rest of sh, the \u and \U
sequences are currently not useful.
Exp-run done by: pav (with some other sh(1) changes)
In particular, this makes things like ${#foo[0]} and ${#foo[@]} errors
rather than silent equivalents of ${#foo}.
PR: bin/151720
Submitted by: Mark Johnston
Exp-run done by: pav (with some other sh(1) changes)
This is only a problem if IFS contains digits, which is unusual but valid.
Because of an incorrect fix for PR bin/12137, "${#parameter}" was treated
as ${#parameter}. The underlying problem was that "${#parameter}"
erroneously added CTLESC bytes before determining the length. This
was properly fixed for PR bin/56147 but the incorrect fix was not backed
out.
Reported by: Seeker on forums.freebsd.org
MFC after: 2 weeks
These already worked: $# ${#} ${##} ${#-} ${#?}
These now work as well: ${#+word} ${#-word} ${##word} ${#%word}
There is an ambiguity in the standard with ${#?}: it could be the length of
$? or it could be $# giving an error in the (impossible) case that it is not
set. We continue to use the former interpretation as it seems more useful.
* In {(...) <redir1;} <redir2, do not drop redir1.
* Maintain the difference between (...) <redir and {(...)} <redir:
In (...) <redir, the redirection is performed in the child, while in
{(...)} <redir it should be performed in the parent (like {(...); :;}
<redir)
* Prefer one CHECKSTRSPACE with multiple USTPUTC to multiple STPUTC.
* Add STPUTS macro (based on function) and use it instead of loops that add
nul-terminated strings to the stack string.
No functional change is intended, but code size is about 1K less on i386.
This moves the function of the noaliases variable into the checkkwd
variable. This way it is properly reset on errors and aliases can be used
normally in the commands for each case (the case labels recognize the
keyword esac but no aliases).
The new code is clearer as well.
Obtained from: dash
These do something else in ksh: name=(...) is an array or compound variable
assignment and the others are extended patterns.
This is the last patch of the ones tested in the exp run.
Exp-run done by: pav (with some other sh(1) changes)
Apart from detecting breakage earlier or at all, this also fixes a segfault
in the testsuite. The "handling" of the breakage left an invalid internal
representation in some cases.
Examples:
echo a; do echo b
echo `) echo a`
echo `date; do do do`
Exp-run done by: pav (with some other sh(1) changes)
The code is inspired by NetBSD sh somewhat, but different because we
preserve the old Almquist/Bourne/Korn ability to have an unquoted part in a
quoted ${v+word}. For example, "${v-"*"}" expands to $v as a single field if
v is set, but generates filenames otherwise.
Note that this is the only place where we split text literally from the
script (the similar ${v=word} assigns to v and then expands $v). The parser
must now add additional markers to allow the expansion code to know whether
arbitrary characters in substitutions are quoted.
Example:
for i in ${$+a b c}; do echo $i; done
Exp-run done by: pav (with some other sh(1) changes)
If double-quote state does not match, treat the '}' literally.
This ensures double-quote state remains the same before and after a
${v+-=?...} which helps with expand.c.
It makes things like
${foo+"\${bar}"}
which I have seen in the wild work as expected.
Exp-run done by: pav (with some other sh(1) changes)
