cfeb7489c2
Add support for kernel fault injection using KFAIL_POINT_* macros and fail_point_* infrastructure. Add example fail point in vfs_bio.c to simulate VM buf pressure. Approved by: dfr (mentor)
198 lines
6.4 KiB
Groff
198 lines
6.4 KiB
Groff
.\"
|
|
.\" Copyright (c) 2009 Isilon Inc http://www.isilon.com/
|
|
.\"
|
|
.\" Redistribution and use in source and binary forms, with or without
|
|
.\" modification, are permitted provided that the following conditions
|
|
.\" are met:
|
|
.\" 1. Redistributions of source code must retain the above copyright
|
|
.\" notice(s), this list of conditions and the following disclaimer as
|
|
.\" the first lines of this file unmodified other than the possible
|
|
.\" addition of one or more copyright notices.
|
|
.\" 2. Redistributions in binary form must reproduce the above copyright
|
|
.\" notice(s), this list of conditions and the following disclaimer in the
|
|
.\" documentation and/or other materials provided with the distribution.
|
|
.\"
|
|
.\" THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY
|
|
.\" EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
|
|
.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
.\" DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY
|
|
.\" DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
|
|
.\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
.\" SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
.\" CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
|
|
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
|
|
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
|
|
.\" DAMAGE.
|
|
.\"
|
|
.\" $FreeBSD$
|
|
.\"
|
|
.Dd May 10, 2009
|
|
.Dt FAIL 9
|
|
.Os
|
|
.Sh NAME
|
|
.Nm KFAIL_POINT_CODE ,
|
|
.Nm KFAIL_POINT_RETURN ,
|
|
.Nm KFAIL_POINT_RETURN_VOID ,
|
|
.Nm KFAIL_POINT_ERROR ,
|
|
.Nm KFAIL_POINT_GOTO ,
|
|
.Nm fail_point ,
|
|
.Nm DEBUG_FP
|
|
.
|
|
.Nd fail points
|
|
.Sh SYNOPSIS
|
|
.In sys/fail.h
|
|
.Fn KFAIL_POINT_CODE "parent" "name" "code"
|
|
.Fn KFAIL_POINT_RETURN "parent" "name"
|
|
.Fn KFAIL_POINT_RETURN_VOID "parent" "name"
|
|
.Fn KFAIL_POINT_ERROR "parent" "name" "error_var"
|
|
.Fn KFAIL_POINT_GOTO "parent" "name" "error_var" "label"
|
|
.Sh DESCRIPTION
|
|
Fail points are used to add code points where errors may be injected
|
|
in a user controlled fashion. Fail points provide a convenient wrapper
|
|
around user provided error injection code, providing a
|
|
.Xr sysctl 9 MIB , and a parser for that MIB that describes how the error
|
|
injection code should fire.
|
|
.Pp
|
|
The base fail point macro is
|
|
.Fn KFAIL_POINT_CODE
|
|
where
|
|
.Fa parent
|
|
is a sysctl tree (frequently
|
|
.Sy DEBUG_FP
|
|
for kernel fail points, but various subsystems may wish to provide
|
|
their own fail point trees), and
|
|
.Fa name
|
|
is the name of the MIB in that tree, and
|
|
.Fa code
|
|
is the error injection code. The
|
|
.Fa code
|
|
argument does not require braces, but it is considered good style to
|
|
use braces for any multi-line code arguments. Inside the
|
|
.Fa code
|
|
argument, the evaluation of
|
|
.Sy RETURN_VALUE
|
|
is derived from the
|
|
.Fn return
|
|
value set in the sysctl MIB. See
|
|
.Sx SYSCTL SETTINGS
|
|
below.
|
|
.Pp
|
|
The remaining
|
|
.Fn KFAIL_POINT_*
|
|
macros are wrappers around common error injection paths:
|
|
.Bl -tag -width 8
|
|
.It Fn KFAIL_POINT_RETURN parent name
|
|
is the equivalent of
|
|
.Sy KFAIL_POINT_CODE(..., return RETURN_VALUE)
|
|
.It Fn KFAIL_POINT_RETURN_VOID parent name
|
|
is the equivalent of
|
|
.Sy KFAIL_POINT_CODE(..., return)
|
|
.It Fn KFAIL_POINT_ERROR parent name error_var
|
|
is the equivalent of
|
|
.Sy KFAIL_POINT_CODE(..., error_var = RETURN_VALUE)
|
|
.It Fn KFAIL_POINT_GOTO parent name error_var label
|
|
is the equivalent of
|
|
.Sy KFAIL_POINT_CODE(...,
|
|
{ error_var = RETURN_VALUE; goto label;})
|
|
.El
|
|
.Pp
|
|
.Sh SYSCTL VARIABLES
|
|
The
|
|
.Fn KFAIL_POINT_*
|
|
macros add sysctl MIBs where specified. Many base kernel MIBs can be
|
|
found in the
|
|
.Sy debug.fail_point
|
|
tree (referenced in code by
|
|
.Sy DEBUG_FP
|
|
).
|
|
.Pp
|
|
The sysctl setting recognizes the following grammar:
|
|
.Pp
|
|
<fail_point> ::
|
|
<term> ( "->" <term> )*
|
|
.Pp
|
|
<term> ::
|
|
( (<float> "%") | (<integer> "*" ) )*
|
|
<type>
|
|
[ "(" <integer> ")" ]
|
|
.Pp
|
|
<float> ::
|
|
<integer> [ "." <integer> ] |
|
|
"." <integer>
|
|
.Pp
|
|
<type> ::
|
|
"off" | "return" | "sleep" | "panic" | "break" | "print"
|
|
.Pp
|
|
The <type>
|
|
argument specifies which action to take:
|
|
.Bl -tag -width ".Dv return"
|
|
.It Sy off
|
|
Take no action (does not trigger fail point code)
|
|
.It Sy return
|
|
Trigger fail point code with specified argument
|
|
.It Sy sleep
|
|
Sleep the specified number of milliseconds
|
|
.It Sy panic
|
|
Panic
|
|
.It Sy break
|
|
Break into the debugger.
|
|
.It Sy print
|
|
Print that the fail point executed
|
|
.El
|
|
.Pp
|
|
The <float>% and <integer>* modifiers prior to <type> control when
|
|
<type> is executed. The <float>% form (e.g. "1.2%") can be used to
|
|
specify a probability that <type> will execute. The <integer>* form
|
|
(e.g. "5*") can be used to specify the number of times <type> should
|
|
be executed before this <term> is disabled. Only the last probability
|
|
and the last count are used if multiple are specified, i.e. "1.2%2%"
|
|
is the same as "2%". When both a probability and a count are
|
|
specified, the probability is evaluated before the count, i.e. "2%5*"
|
|
means "2% of the time, but only execute it 5 times total".
|
|
.Pp
|
|
The operator -> can be used to express cascading terms. If you specify
|
|
<term1>-><term2>, it means that if <term1> doesn't 'execute', <term2>
|
|
is evaluated. For the purpose of this operator, the return() and
|
|
print() operators are the only types that cascade. A return() term
|
|
only cascades if the code executes, and a print() term only cascades
|
|
when passed a non-zero argument.
|
|
.Pp
|
|
.Sh EXAMPLES
|
|
.Bl -tag
|
|
.It Sy sysctl debug.fail_point.foobar="2.1%return(5)"
|
|
21/1000ths of the time, execute
|
|
.Fa code
|
|
with RETURN_VALUE set to 5.
|
|
.It Sy sysctl debug.fail_point.foobar="2%return(5)->5%return(22)"
|
|
2/100th of the time, execute
|
|
.Fa code
|
|
with RETURN_VALUE set to 5. If that doesn't happen, 5% of the time
|
|
execute
|
|
.Fa code
|
|
with RETURN_VALUE set to 22.
|
|
.It Sy sysctl debug.fail_point.foobar="5*return(5)->0.1%return(22)"
|
|
For 5 times, return 5. After that, 1/1000ths of the time, return 22.
|
|
.It Sy sysctl debug.fail_point.foobar="0.1%5*return(5)"
|
|
Return 5 for 1 in 1000 executions, but only execute 5 times total.
|
|
.It Sy sysctl debug.fail_point.foobar="1%*sleep(50)"
|
|
1/100ths of the time, sleep 50ms.
|
|
.El
|
|
.Pp
|
|
.Sh CAVEATS
|
|
It's easy to shoot yourself in the foot by setting fail points too
|
|
aggressively or setting too many in combination. For example, forcing
|
|
.Fn malloc
|
|
to fail consistently is potentially harmful to uptime.
|
|
.Pp
|
|
The
|
|
.Fn sleep
|
|
sysctl setting may not be appropriate in all situations. Currently,
|
|
.Fn fail_point_eval
|
|
does not verify whether the context is appropriate for calling
|
|
.Fn msleep .
|
|
.Pp
|
|
.Sh AUTHORS
|
|
.An -nosplit
|
|
This manual page was written by
|
|
.An Zach Loafman Aq zml@FreeBSD.org .
|