Commit Graph

125 Commits

Author SHA1 Message Date
jilles
6fa139d56e sh: Expand here documents in the current process.
Expand here documents at the same point other redirections are expanded but
use a non-fork subshell environment (like simple command substitutions) for
compatibility. Substitition errors result in an empty here document like
before.

As a result, a fork is avoided for short (<4K) expanded here documents.

Unexpanded here documents (with quoted end marker after <<) are not affected
by this change. They already only forked when >4K.

Side effects:
* Order of expansion is slightly different.
* Slow expansions are not executed in parallel with the redirected command.
* A non-fork subshell environment is subtly different from a forked process.
2013-02-03 15:54:57 +00:00
jilles
81fe037601 sh: Move some stackmarks to fix high memory usage in some loops.
If a loop contained certain commands (such as redirected compound commands),
the temporary memory for the redirection was not freed between iterations of
the loop but only after the loop.

Put a stackmark in evaltree(), freeing memory whenever a node has been
evaluated. Some other stackmarks are then redundant; remove them.

Example:
  while :; do { :; } </dev/null; done
2013-01-20 21:28:05 +00:00
jilles
d139340997 sh: Pass $? to command substitution containing compound/multiple commands.
Example:
  false; echo $(echo $?; :)
2013-01-14 12:20:55 +00:00
jilles
e5dd01ef24 sh: Detect and flag write errors on stdout in builtins.
If there is a write error on stdout, a message will be printed (to stderr)
and the exit status will be changed to 2 if it would have been 0 or 1.

PR:		bin/158206
2012-12-12 22:01:10 +00:00
jilles
99ca87dd2d sh: Prefer internal nextopt() to libc getopt().
This reduces code duplication and code size.

/usr/bin/printf is not affected.

Side effect: different error messages when certain builtins are passed
invalid options.
2012-09-15 21:56:30 +00:00
jilles
689774f8e7 sh: Expand assignment-like words specially for export/readonly/local.
Examples:
  export x=~
now expands the tilde
  local y=$1
is now safe, even if $1 contains IFS characters or metacharacters.

For a word to "look like an assignment", it must start with a name followed
by an equals sign, none of which may be quoted.

The special treatment applies when the first word (potentially after
"command") is "export", "readonly" or "local". There may be quoting
characters but no expansions. If "local" is overridden with a function there
is no special treatment ("export" and "readonly" cannot be overridden with a
function).

If things like
  local arr=(1 2 3)
are ever allowed in the future, they cannot call a "local" function. This
would either be a run-time error or it would call the builtin.

This matches Austin Group bug #351, planned for the next issue of POSIX.1.

PR:		bin/166771
2012-07-15 10:19:43 +00:00
jilles
c9a60ad55a sh: Use vfork in a few common cases.
This uses vfork() for simple commands and command substitutions containing a
single simple command, invoking an external program under certain conditions
(no redirections or variable assignments, non-interactive shell, no job
control). These restrictions limit the amount of code executed in a vforked
child.

There is a large speedup (for example 35%) in microbenchmarks. The
difference in buildkernel is smaller (for example 0.5%) but still
statistically significant. See
http://lists.freebsd.org/pipermail/freebsd-hackers/2012-January/037581.html
for some numbers.

The use of vfork() can be disabled by setting a variable named
SH_DISABLE_VFORK.
2012-02-04 23:12:14 +00:00
jilles
ae59680813 sh: Fix $? in the first command of a 'for'.
In the first command of a 'for', $? should be the exit status of the last
pipeline (command substitution in the word list or command before 'for'),
not always 0.
2012-01-22 14:00:33 +00:00
dumbbell
8b387a2d35 sh: Fix execution of multiple statements in a trap when evalskip is set
Before this fix, only the first statement of the trap was executed if
evalskip was set. This is for example the case when:
    o  "-e" is set for this shell
    o  a trap is set on EXIT
    o  a function returns 1 and causes the script to abort

Reviewed by:	jilles
MFC after:	2 weeks
2012-01-16 11:07:46 +00:00
jilles
0ae130814a sh: Fix some bugs with exit status from case containing ;&.
Also, rework evalcase() to not evaluate any tree. Instead, return the
NCLISTFALLTHRU node and handle it in evaltree().

Fixed bugs:

* If a ;& list with non-zero exit status is followed by an empty ;; or final
  list, the exit status of the case command should be equal to the exit
  status of the ;& list, not 0.

* An empty ;& case should not reset $?.
2012-01-15 21:39:38 +00:00
jilles
6253417a70 sh: Fix two bugs with case and exit status:
* If no pattern is matched, POSIX says the exit status shall be 0 (even if
  there are command substitutions).
* If a pattern is matched and there are no command substitutions, the first
  command should see the $? from before the case command, not always 0.
2012-01-15 20:04:05 +00:00
jilles
45b77afcc1 sh: Do not force special builtins non-special in optimized command subst.
This is not necessary: errors are already caught in evalbackcmd() and
forcelocal handles changes to variables.

Note that this depends on r223024.

MFC after:	4 weeks
2011-12-28 22:10:12 +00:00
jilles
519601e533 sh: Remove impossible evalskip check in 'for'. 2011-11-27 00:09:59 +00:00
jilles
49f86abcab sh: Reduce one level of evaltree() recursion when executing 'case'.
Free expanded case text before executing commands.
Remove impossible evalskip checks (expanding an argument cannot set
evalskip anymore since $(break) and the like are properly executed in a
subshell environment).
2011-11-26 23:28:31 +00:00
jilles
f78ebc168b sh: Remove special support for background simple commands.
It expands the arguments in the parent shell process, which is incorrect.
2011-06-18 23:58:59 +00:00
jilles
c7a72567a8 sh: Add case statement fallthrough (with ';&' instead of ';;').
Replacing ;; with the new control operator ;& will cause the next list to be
executed as well without checking its pattern, continuing until a list ends
with ;; or until the end of the case statement. This is like omitting
"break" in a C "switch" statement.

The sequence ;& was formerly invalid.

This feature is proposed for the next POSIX issue in Austin Group issue
#449.
2011-06-17 13:03:49 +00:00
jilles
00d33feb9d sh: Reduce unnecessary forks with eval.
The eval special builtin now runs the code with EV_EXIT if it was run
with EV_EXIT itself.

In particular, this eliminates one fork when a command substitution contains
an eval command that ends with an external program or a subshell.

This is similar to what r220978 did for functions.
2011-06-16 21:50:28 +00:00
jilles
91789615b4 sh: Save/restore changed variables in optimized command substitution.
In optimized command substitution, save and restore any variables changed by
expansions (${var=value} and $((var=assigned))), instead of trying to
determine if an expansion may cause such changes.

If $! is referenced in optimized command substitution, do not cause jobs to
be remembered longer.

This fixes $(jobs $!) again, simplifies the man page and shortens the code.
2011-06-12 23:06:04 +00:00
jilles
bd55770c94 sh: Do parameter expansion before printing PS4 (set -x).
The function name expandstr() and the general idea of doing this kind of
expansion by treating the text as a here document without end marker is from
dash.

All variants of parameter expansion and arithmetic expansion also work (the
latter is not required by POSIX but it does not take extra code and many
other shells also allow it).

Command substitution is prevented because I think it causes too much code to
be re-entered (for example creating an unbounded recursion of trace lines).

Unfortunately, our LINENO is somewhat crude, otherwise PS4='$LINENO+ ' would
be quite useful.
2011-06-09 23:12:23 +00:00
jilles
fba76e8544 sh: Fix $? in heredocs on simple commands.
PR:		bin/41410
2011-06-05 14:13:15 +00:00
jilles
0383d0751d sh: Honour -n while processing -c string. 2011-06-04 11:28:42 +00:00
jilles
3dd8ae4222 sh: Expand aliases after assignments and redirections. 2011-05-21 22:03:06 +00:00
jilles
ee209e5ce8 sh: Avoid close(-1) when evaluating a multi-command pipeline.
Valgrind complains about this.
2011-05-15 17:00:43 +00:00
jilles
7b50330e01 sh: Set $? to 0 for background commands.
For backgrounded pipelines and subshells, the previous value of $? was being
preserved, which is incorrect.

For backgrounded simple commands containing a command substitution, the
status of the last command substitution was returned instead of 0.

If fork() fails, this is an error.
2011-04-25 20:54:12 +00:00
jilles
f250dc2f44 sh: Allow EV_EXIT through function calls, make {...} <redir more consistent.
If EV_EXIT causes an exit, use the exception mechanism to unwind
redirections and local variables. This way, if the final command is a
redirected command, an EXIT trap now executes without the redirections.

Because of these changes, EV_EXIT can now be inherited by the body of a
function, so do so. This means that a function no longer prevents a fork
before an exec being skipped, such as in
  f() { head -1 /etc/passwd; }; echo $(f)

Wrapping a single builtin in a function may still cause an otherwise
unnecessary fork with command substitution, however.

An exit command or -e failure still invokes the EXIT trap with the
original redirections and local variables in place.

Note: this depends on SHELLPROC being gone. A SHELLPROC depended on
keeping the redirections and local variables and only cleaning up the
state to restore them.
2011-04-23 22:28:56 +00:00
jilles
9a75a8c404 sh: Remove clearcmdentry()'s now unused argument. 2011-02-05 14:08:51 +00:00
jilles
95ad413d4a sh: Remove special code for shell scripts without magic number.
These are called "shell procedures" in the source.

If execve() failed with [ENOEXEC], the shell would reinitialize itself
and execute the program as a script. This requires a fair amount of code
which is not frequently used (most scripts have a #! magic number).
Therefore just execute a new instance of sh (_PATH_BSHELL) to run the
script.
2011-02-04 22:47:55 +00:00
jilles
3c4cff0f35 sh: Do not call exitshell() from evalcommand() unless evalcommand() forked
itself.

This ensures that certain traps caused by builtins are executed.
2011-01-05 23:17:29 +00:00
jilles
9391068711 sh: Check readonly status for assignments on regular builtins.
An error message is written, the builtin is not executed, nonzero exit
status is returned but the shell does not abort.

This was already checked for special builtins and external commands, with
the same consequences except that the shell aborts for special builtins.

Obtained from:	NetBSD
2011-01-01 13:26:18 +00:00
jilles
ca3118f4ca sh: Avoid side effects from builtins in optimized command substitution.
Change the criterion for builtins to be safe to execute in the same process
in optimized command substitution from a blacklist of only cd, . and eval to
a whitelist.

This avoids clobbering the main shell environment such as by $(exit 4) and
$(set -x).

The builtins jobid, jobs, times and trap can still show information not
available in a child process; this is deliberately permitted. (Changing
traps is not.)

For some builtins, whether they are safe depends on the arguments passed to
them. Some of these are always considered unsafe to keep things simple; this
only harms efficiency a little in the rare case they are used alone in a
command substitution.
2010-12-30 22:33:55 +00:00
jilles
74d9b02bb0 sh: Don't do optimized command substitution if expansions have side effects.
Before considering to execute a command substitution in the same process,
check if any of the expansions may have a side effect; if so, execute it in
a new process just like happens if it is not a single simple command.

Although the check happens at run time, it is a static check that does not
depend on current state. It is triggered by:
- expanding $! (which may cause the job to be remembered)
- ${var=value} default value assignment
- assignment operators in arithmetic
- parameter substitutions in arithmetic except ${#param}, $$, $# and $?
- command substitutions in arithmetic

This means that $((v+1)) does not prevent optimized command substitution,
whereas $(($v+1)) does, because $v might expand to something containing
assignment operators.

Scripts should not depend on these exact details for correctness. It is also
imaginable to have the shell fork if and when a side effect is encountered
or to create a new temporary namespace for variables.

Due to the $! change, the construct $(jobs $!) no longer works. The value of
$! should be stored in a variable outside command substitution first.
2010-12-28 21:27:08 +00:00
jilles
713ef02a1f sh: Make expansion errors in optimized command substitution non-fatal.
Command substitutions consisting of a single simple command are executed in
the main shell process but this should be invisible apart from performance
and very few exceptions such as $(trap).
2010-12-28 13:28:24 +00:00
jilles
7377de8f91 sh: Code size optimizations to "stack string" memory allocation:
* Prefer one CHECKSTRSPACE with multiple USTPUTC to multiple STPUTC.
* Add STPUTS macro (based on function) and use it instead of loops that add
  nul-terminated strings to the stack string.

No functional change is intended, but code size is about 1K less on i386.
2010-11-23 22:17:39 +00:00
jilles
6915411ab2 sh: Code size optimizations to buffered output.
This is mainly less use of the outc macro.

No functional change is intended, but code size is about 2K less on i386.
2010-11-20 14:14:52 +00:00
jilles
1685738e37 sh: Reindent evaltree(). 2010-10-31 12:08:16 +00:00
jilles
4de067d3c2 sh: Use iteration instead of recursion to evaluate semicolon lists.
This reduces CPU and memory usage when executing long lists (such
as long functions).
2010-10-31 12:06:02 +00:00
jilles
2ae15286ba sh: Tweak some string constants to reduce code size.
* Reduce some needless differences.
* Shorten some error messages that should not happen.
2010-10-29 21:44:43 +00:00
obrien
08b8d916b5 In the spirit of r90111, depend on c89 and remove the "STATIC" macro
and its usage.
2010-10-13 22:18:03 +00:00
obrien
f31ad1c86b Consistently use "STATIC" for all functions in order to be able to set
breakpoints with in a debugger.  And use naked "static" for variables.

Noticed by:	bde
2010-10-13 04:01:01 +00:00
jilles
2beda3228f sh: Fix exit status if return is used within a loop condition. 2010-09-11 15:07:40 +00:00
jilles
0f8d870bb8 sh: Get rid of some magic numbers.
MFC after:	1 week
2010-09-04 21:23:46 +00:00
jilles
d713120364 sh: Add a forgotten const. 2010-08-13 20:29:43 +00:00
jilles
184699830c sh: Return 0 from eval if no command was given.
This makes a difference if there is a command substitution.

To make this work, evalstring() has been changed to set exitstatus to 0 if
no command was executed (the string contained only whitespace).

Example:
  eval $(false); echo $?
should print 0.
2010-08-03 22:17:29 +00:00
jilles
c5fcbff43a sh: Recognize "--" in . and exec.
Although "--" historically has not been required to be recognized for
certain special builtins that do not take options in POSIX, some other
implementations recognize options for them, requiring scripts to use "--" or
avoid operands starting with "-".

Operands starting with "-" can be avoided with eval by prepending a space,
and cannot occur with break, continue, exit, return and shift as they only
take numbers, nor with times as it does not take operands. With . and exec,
avoiding "-" is not so easy as it may require reimplementing the PATH
search; therefore the current proposal for POSIX is to require recognition
of "--" for them.

We continue to accept other strings starting with "-" as operands to . and
exec, and also "--" if it is alone to . (which would otherwise be invalid
anyway).
2010-05-28 22:40:24 +00:00
jilles
6a8de408d7 sh: Have only one copy of _PATH_STDPATH in the binary. 2010-05-08 14:00:01 +00:00
jilles
f3856c6cf2 sh: Apply locale vars on builtins, recognize LC_MESSAGES as a locale var.
This allows doing things like LC_ALL=C some_builtin to run a builtin under a
different locale, just like is possible with external programs. The
immediate reason is that this allows making printf(1) a builtin without
breaking things like LC_NUMERIC=C printf '%f\n' 1.2

This change also affects special builtins, as even though the assignment is
persistent, the export is only to the builtin (unless the variable was
already exported).

Note: for this to work for builtins that also exist as external programs
such as /bin/test, the setlocale() call must be under #ifndef SHELL. The
shell will do the setlocale() calls which may not agree with the environment
variables.
2010-05-05 21:48:40 +00:00
jilles
d88c6b2784 sh: Do not abort on a redirection error on a compound command.
Redirection errors on subshells already did not abort the shell because
the redirection is executed in the subshell.

Other shells seem to agree that these redirection errors should not abort
the shell.

Also ensure that the redirections will be cleaned up properly in cases like
  command eval '{ shift x; } 2>/dev/null'

Example:
  { echo bad; } </var/empty/x; echo good
2010-03-14 14:24:35 +00:00
jilles
6be35ddc1a sh: Do not abort on a redirection error if there is no command word.
Although simple commands without a command word (only assignments and/or
redirections) are much like special builtins, POSIX and most shells seem to
agree that redirection errors should not abort the shell in this case. Of
course, the assignments persist and assignment errors are fatal.

To get the old behaviour portably, use the ':' special builtin.
To get the new behaviour portably, given that there are no assignments, use
the 'true' regular builtin.
2010-03-13 22:53:17 +00:00
jilles
3242a81cf4 sh: Make sure to popredir() even if a function caused an error. 2010-03-06 17:31:09 +00:00
jilles
c30671a949 sh: Make sure to popredir() even if a special builtin caused an error. 2010-03-06 17:09:22 +00:00