Move dynamic sysctl(8) variable creation for the cd(4) and da(4) drivers

out of cdregister() and daregister(), which are run from interrupt context.

The sysctl code does blocking mallocs (M_WAITOK), which causes problems
if malloc(9) actually needs to sleep.

The eventual fix for this issue will involve moving the CAM probe process
inside a kernel thread.  For now, though, I have fixed the issue by moving
dynamic sysctl variable creation for these two drivers to a task queue
running in a kernel thread.

The existing task queues (taskqueue_swi and taskqueue_swi_giant) run in
software interrupt handlers, which wouldn't fix the problem at hand.  So I
have created a new task queue, taskqueue_thread, that runs inside a kernel
thread.  (It also runs outside of Giant -- clients must explicitly acquire
and release Giant in their taskqueue functions.)

scsi_cd.c:	Remove sysctl variable creation code from cdregister(), and
		move it to a new function, cdsysctlinit().  Queue
		cdsysctlinit() to the taskqueue_thread taskqueue once we
		have fully registered the cd(4) driver instance.

scsi_da.c:	Remove sysctl variable creation code from daregister(), and
		move it to move it to a new function, dasysctlinit().
		Queue dasysctlinit() to the taskqueue_thread taskqueue once
		we have fully registered the da(4) instance.

taskqueue.h:	Declare the new taskqueue_thread taskqueue, update some
		comments.

subr_taskqueue.c:
		Create the new kernel thread taskqueue.  This taskqueue
		runs outside of Giant, so any functions queued to it would
		need to explicitly acquire/release Giant if they need it.

cd.4:		Update the cd(4) man page to talk about the minimum command
		size sysctl/loader tunable.  Also note that the changer
		variables are available as loader tunables as well.

da.4:		Update the da(4) man page to cover the retry_count,
		default_timeout and minimum_cmd_size sysctl variables/loader
		tunables.  Remove references to /dev/r???, they aren't used
		any longer.

cd.9:		Update the cd(9) man page to describe the CD_Q_10_BYTE_ONLY
		quirk.

taskqueue.9:	Update the taskqueue(9) man page to describe the new thread
		task queue, and the taskqueue_swi_giant queue.

MFC after:	3 days
This commit is contained in:
Kenneth D. Merry 2003-09-03 04:46:28 +00:00
parent cfd5600c66
commit cb32189e23
8 changed files with 266 additions and 101 deletions

View File

@ -25,7 +25,7 @@
.\"
.\" $FreeBSD$
.\"
.Dd October 10, 1998
.Dd September 2, 2003
.Dt CD 4
.Os
.Sh NAME
@ -370,6 +370,37 @@ Some work is planned to support
some of the more common `broken'
.Tn CD-ROM
drives; however, this is not yet under way.
.Pp
The
.Nm
driver attempts to automatically determine whether the drive it is talking
to supports 6 byte or 10 byte MODE SENSE/MODE SELECT operations. Many
.Tn SCSI
drives only support 6 byte commands, and
.Tn ATAPI
drives only support 10 byte commands.
The
.Nm
driver first attempts to determine whether the protocol in use typically
supports 6 byte commands by issuing a CAM Path Inquiry CCB.
It will then default to 6 byte or 10 byte commands as appropriate.
After that, the
.Nm
driver defaults to using 6 byte commands (assuming the protocol the drive
speaks claims to support 6 byte commands), until one fails with a
.Tn SCSI
ILLEGAL REQUEST error. Then it tries the 10 byte version of the command to
see if that works instead. Users can change the default via per-drive
sysctl variables and loader tunables. The variable names are the same in
both instances:
.Pp
.Va kern.cam.cd.%d.minimum_cmd_size
.Pp
Where
.Dq %d
is the unit number of the drive in question. Valid minimum command sizes
are 6 and 10. Any value above 6 will be rounded to 10, and any value below
6 will be rounded to 6.
.Sh CHANGER OPERATION
This driver has built-in support for LUN-based CD changers.
A LUN-based CD
@ -399,7 +430,7 @@ If there is no outstanding I/O for
another LUN, the driver will allow indefinite access to a given LUN.
.Pp
The minimum and maximum time quanta are configurable via kernel options and
also via sysctl variables.
also via sysctl and kernel tunable variables.
The kernel options are:
.Pp
.Bl -item -compact
@ -409,7 +440,7 @@ The kernel options are:
.Cd "options ""CHANGER_MAX_BUSY_SECONDS=11"""
.El
.Pp
The sysctl variables are:
The sysctl/kernel tunable variables are:
.Pp
.Bl -item -compact
.It

View File

@ -25,7 +25,7 @@
.\"
.\" $FreeBSD$
.\"
.Dd October 15, 1998
.Dd September 2, 2003
.Dt DA 4
.Os
.Sh NAME
@ -196,6 +196,59 @@ The driver
.Em will
write the new disklabel to the disk.
.El
.Sh SYSCTL VARIABLES
The following variables are available as both
.Xr sysctl 8
variables and
.Xr loader 8
tunables:
.Bl -tag -width 12
.It kern.cam.da.retry_count
.Pp
This variable determines how many times the
.Nm
driver will retry a READ or WRITE command.
This does not affect the number of retries used during probe time or for
the
.Nm
driver dump routine.
This value currently defaults to 4.
.It kern.cam.da.default_timeout
.Pp
This variable determines how long the
.Nm
driver will wait before timing out an outstanding command.
The units for this value are seconds, and the default is currently 60
seconds.
.It kern.cam.da.%d.minimum_cmd_size
.Pp
This variable determines what the minimum READ/WRITE CDB size is for a
given
.Nm
unit.
(The %d above denotes the unit number of the
.Nm
driver instance, e.g. 1, 2, 4, 8, etc.)
Valid minimum command size values are 6, 10, 12 and 16 bytes.
The default is 6 bytes.
.Pp
The
.Nm
driver issues a CAM Path Inquiry CCB at probe time to determine whether the
protocol the device in question speaks (e.g. ATAPI) typically doesn't allow
6 byte commands.
If it doesn't, the
.Nm
driver will default to using at least 10 byte CDBs.
If a 6 byte READ or WRITE fails with an ILLEGAL REQUEST error, the
.Nm
driver will then increase the default CDB size for the device to 10 bytes and
retry the command.
CDB size is always
chosen as the smallest READ/WRITE CDB that will satisfy the specified minimum
command size, and the LBA and length of the READ or WRITE in question.
(e.g., a write to an LBA larger than 2^32 will require a 16 byte CDB.)
.El
.Sh NOTES
If a device becomes invalidated (media is removed, device becomes unresponsive)
the disklabel and information held within the kernel about the device will
@ -206,25 +259,9 @@ the last file descriptor referencing the old device is closed.
During this period, all new open attempts will be rejected.
.Sh FILES
.Bl -tag -width /dev/rsdXXXXX -compact
.It Pa /dev/rda Ns Ar u
raw mode
.Tn SCSI
disk unit
.Ar u ,
accessed as an unpartitioned device
.Sm off
.It Pa /dev/da Ar u Pa s Ar n
.Sm on
block mode
.Tn SCSI
disk unit
.Ar u ,
slice
.Ar n ,
accessed as an unpartitioned device
.Sm off
.It Pa /dev/rda Ar u Pa s Ar n
.Sm on
raw mode
.Tn SCSI
disk unit
@ -233,15 +270,6 @@ slice
.Ar n ,
accessed as an unpartitioned device
.It Pa /dev/da Ns Ar u Ns Ar p
block mode
.Tn SCSI
disk unit
.Ar u ,
first
.Fx
slice, partition
.Ar p
.It Pa /dev/rda Ns Ar u Ns Ar p
raw mode
.Tn SCSI
disk unit
@ -259,22 +287,6 @@ slice, partition
.Ar p
.Xc
.Sm on
block mode
.Tn SCSI
disk unit
.Ar u ,
.Ar n Ns th
slice, partition
.Ar p
.Sm off
.It Xo
.Pa /dev/rda
.Ar u
.Pa s
.Ar n
.Ar p
.Xc
.Sm on
raw mode
.Tn SCSI
disk unit

View File

@ -27,7 +27,7 @@
.\"
.\" $FreeBSD$
.\"
.Dd October 10, 1998
.Dd September 2, 2003
.Dt CD 9
.Os
.Sh NAME
@ -74,6 +74,22 @@ In general, the driver will figure this out automatically when it sees a
LUN greater than 0. Setting this flag only has the effect of telling the
driver to run the initial read capacity command for LUN 0 of the changer
through the changer scheduling code.
.It Dv CD_Q_10_BYTE_ONLY
This flag tells the driver that the given device only accepts 10 byte MODE
SENSE/MODE SELECT commands. In general these types of quirks should not be
added to the
.Xr cd 4
driver. The reason is that the driver does several things to attempt to
determine whether the drive in question needs 10 byte commands. First, it
issues a CAM Path Inquiry command to determine whether the protocol that
the drive speaks typically only allows 10 byte commands. (ATAPI and USB
are two prominent examples of protocols where you generally only want to
send 10 byte commands.) Then, if it gets an ILLEGAL REQUEST error back
from a 6 byte MODE SENSE or MODE SELECT command, it attempts to send the 10
byte version of the command instead. The only reason you would need a
quirk is if your drive uses a protocol (e.g.
.Tn SCSI )
that typically doesn't have a problem with 6 byte commands.
.El
.Sh FILES
.Bl -tag -width /sys/cam/scsi/scsi_cd.c -compact

View File

@ -174,26 +174,43 @@ argument to the macro is executed as a C statement,
allowing any further initialisation to be performed
(such as registering an interrupt handler etc.)
.Pp
The system provides a global taskqueue,
The system provides three global taskqueues,
.Va taskqueue_swi ,
which is run via a software interrupt mechanism.
To use this queue,
.Va taskqueue_swi_giant ,
and
.Va taskqueue_thread .
The swi taskqueues are run via a software interrupt mechanism.
The taskqueue_swi queue runs without the protection of the Giant kernel lock,
and the taskqueue_swi_giant queue runs with the protection of the Giant
kernel lock.
The thread taskqueue runs in a kernel thread context, and tasks run from
this thread do not run under the Giant kernel lock.
If the caller wants to run under Giant, he should explicitly acquire and
release Giant in his taskqueue handler routine.
To use these queues,
call
.Fn taskqueue_enqueue
with the value of the global variable
.Va taskqueue_swi .
The queue will be run at
.\" XXX This should be a cross-reference (Xr), but there is no MANLINKS
.\" entry for splsofttq.9 yet.
.Fn splsofttq .
with the value of the global taskqueue variable for the queue you wish to
use (
.Va taskqueue_swi ,
.Va taskqueue_swi_giant ,
or
.Va taskqueue_thread
).
.Pp
This queue can be used,
This the software interrupt queues can be used,
for instance, for implementing interrupt handlers which must perform a
significant amount of processing in the handler.
The hardware interrupt handler would perform minimal processing of the
interrupt and then enqueue a task to finish the work.
This reduces to a minimum
the amount of time spent with interrupts disabled.
.Pp
The thread queue can be used, for instance, by interrupt level routines
that need to call kernel functions that do things that can only be done
from a thread context.
(e.g., call malloc with the M_WAITOK flag.)
.Sh HISTORY
This interface first appeared in
.Fx 5.0 .

View File

@ -62,6 +62,7 @@ __FBSDID("$FreeBSD$");
#include <sys/dvdio.h>
#include <sys/devicestat.h>
#include <sys/sysctl.h>
#include <sys/taskqueue.h>
#include <cam/cam.h>
#include <cam/cam_ccb.h>
@ -154,6 +155,7 @@ struct cd_softc {
eventhandler_tag clonetag;
int minimum_command_size;
int outstanding_cmds;
struct task sysctl_task;
struct sysctl_ctx_list sysctl_ctx;
struct sysctl_oid *sysctl_tree;
STAILQ_HEAD(, cd_mode_params) mode_queue;
@ -598,6 +600,43 @@ cdasync(void *callback_arg, u_int32_t code,
}
}
static void
cdsysctlinit(void *context, int pending)
{
struct cam_periph *periph;
struct cd_softc *softc;
char tmpstr[80], tmpstr2[80];
periph = (struct cam_periph *)context;
softc = (struct cd_softc *)periph->softc;
snprintf(tmpstr, sizeof(tmpstr), "CAM CD unit %d", periph->unit_number);
snprintf(tmpstr2, sizeof(tmpstr2), "%d", periph->unit_number);
mtx_lock(&Giant);
sysctl_ctx_init(&softc->sysctl_ctx);
softc->sysctl_tree = SYSCTL_ADD_NODE(&softc->sysctl_ctx,
SYSCTL_STATIC_CHILDREN(_kern_cam_cd), OID_AUTO,
tmpstr2, CTLFLAG_RD, 0, tmpstr);
if (softc->sysctl_tree == NULL) {
printf("cdsysctlinit: unable to allocate sysctl tree\n");
return;
}
/*
* Now register the sysctl handler, so the user can the value on
* the fly.
*/
SYSCTL_ADD_PROC(&softc->sysctl_ctx,SYSCTL_CHILDREN(softc->sysctl_tree),
OID_AUTO, "minimum_cmd_size", CTLTYPE_INT | CTLFLAG_RW,
&softc->minimum_command_size, 0, cdcmdsizesysctl, "I",
"Minimum CDB size");
mtx_unlock(&Giant);
}
/*
* We have a handler function for this so we can check the values when the
* user sets them, instead of every time we look at them.
@ -642,7 +681,7 @@ cdregister(struct cam_periph *periph, void *arg)
struct ccb_setasync csa;
struct ccb_pathinq cpi;
struct ccb_getdev *cgd;
char tmpstr[80], tmpstr2[80];
char tmpstr[80];
caddr_t match;
cgd = (struct ccb_getdev *)arg;
@ -696,17 +735,7 @@ cdregister(struct cam_periph *periph, void *arg)
if (cpi.ccb_h.status == CAM_REQ_CMP && (cpi.hba_misc & PIM_NO_6_BYTE))
softc->quirks |= CD_Q_10_BYTE_ONLY;
snprintf(tmpstr, sizeof(tmpstr), "CAM CD unit %d", periph->unit_number);
snprintf(tmpstr2, sizeof(tmpstr2), "%d", periph->unit_number);
sysctl_ctx_init(&softc->sysctl_ctx);
softc->sysctl_tree = SYSCTL_ADD_NODE(&softc->sysctl_ctx,
SYSCTL_STATIC_CHILDREN(_kern_cam_cd), OID_AUTO,
tmpstr2, CTLFLAG_RD, 0, tmpstr);
if (softc->sysctl_tree == NULL) {
printf("cdregister: unable to allocate sysctl tree\n");
free(softc, M_DEVBUF);
return (CAM_REQ_CMP_ERR);
}
TASK_INIT(&softc->sysctl_task, 0, cdsysctlinit, periph);
/* The default is 6 byte commands, unless quirked otherwise */
if (softc->quirks & CD_Q_10_BYTE_ONLY)
@ -727,15 +756,6 @@ cdregister(struct cam_periph *periph, void *arg)
else if (softc->minimum_command_size > 6)
softc->minimum_command_size = 10;
/*
* Now register the sysctl handler, so the user can the value on
* the fly.
*/
SYSCTL_ADD_PROC(&softc->sysctl_ctx,SYSCTL_CHILDREN(softc->sysctl_tree),
OID_AUTO, "minimum_cmd_size", CTLTYPE_INT | CTLFLAG_RW,
&softc->minimum_command_size, 0, cdcmdsizesysctl, "I",
"Minimum CDB size");
/*
* We need to register the statistics structure for this device,
* but we don't have the blocksize yet for it. So, we register
@ -1847,6 +1867,11 @@ cddone(struct cam_periph *periph, union ccb *done_ccb)
xpt_announce_periph(periph, announce_buf);
if (softc->flags & CD_FLAG_CHANGER)
cdchangerschedule(softc);
/*
* Create our sysctl variables, now that we know
* we have successfully attached.
*/
taskqueue_enqueue(taskqueue_thread,&softc->sysctl_task);
}
softc->state = CD_STATE_NORMAL;
/*

View File

@ -48,6 +48,7 @@ __FBSDID("$FreeBSD$");
#include <sys/eventhandler.h>
#include <sys/malloc.h>
#include <sys/cons.h>
#include <sys/taskqueue.h>
#include <machine/md_var.h>
@ -133,6 +134,7 @@ struct da_softc {
struct disk_params params;
struct disk disk;
union ccb saved_ccb;
struct task sysctl_task;
struct sysctl_ctx_list sysctl_ctx;
struct sysctl_oid *sysctl_tree;
};
@ -388,6 +390,7 @@ static dumper_t dadump;
static periph_init_t dainit;
static void daasync(void *callback_arg, u_int32_t code,
struct cam_path *path, void *arg);
static void dasysctlinit(void *context, int pending);
static int dacmdsizesysctl(SYSCTL_HANDLER_ARGS);
static periph_ctor_t daregister;
static periph_dtor_t dacleanup;
@ -915,6 +918,41 @@ daasync(void *callback_arg, u_int32_t code,
}
}
static void
dasysctlinit(void *context, int pending)
{
struct cam_periph *periph;
struct da_softc *softc;
char tmpstr[80], tmpstr2[80];
periph = (struct cam_periph *)context;
softc = (struct da_softc *)periph->softc;
snprintf(tmpstr, sizeof(tmpstr), "CAM DA unit %d", periph->unit_number);
snprintf(tmpstr2, sizeof(tmpstr2), "%d", periph->unit_number);
mtx_lock(&Giant);
sysctl_ctx_init(&softc->sysctl_ctx);
softc->sysctl_tree = SYSCTL_ADD_NODE(&softc->sysctl_ctx,
SYSCTL_STATIC_CHILDREN(_kern_cam_da), OID_AUTO, tmpstr2,
CTLFLAG_RD, 0, tmpstr);
if (softc->sysctl_tree == NULL) {
printf("dasysctlinit: unable to allocate sysctl tree\n");
return;
}
/*
* Now register the sysctl handler, so the user can the value on
* the fly.
*/
SYSCTL_ADD_PROC(&softc->sysctl_ctx,SYSCTL_CHILDREN(softc->sysctl_tree),
OID_AUTO, "minimum_cmd_size", CTLTYPE_INT | CTLFLAG_RW,
&softc->minimum_cmd_size, 0, dacmdsizesysctl, "I",
"Minimum CDB size");
mtx_unlock(&Giant);
}
static int
dacmdsizesysctl(SYSCTL_HANDLER_ARGS)
{
@ -955,7 +993,7 @@ daregister(struct cam_periph *periph, void *arg)
struct ccb_setasync csa;
struct ccb_pathinq cpi;
struct ccb_getdev *cgd;
char tmpstr[80], tmpstr2[80];
char tmpstr[80];
caddr_t match;
cgd = (struct ccb_getdev *)arg;
@ -1008,17 +1046,7 @@ daregister(struct cam_periph *periph, void *arg)
if (cpi.ccb_h.status == CAM_REQ_CMP && (cpi.hba_misc & PIM_NO_6_BYTE))
softc->quirks |= DA_Q_NO_6_BYTE;
snprintf(tmpstr, sizeof(tmpstr), "CAM DA unit %d", periph->unit_number);
snprintf(tmpstr2, sizeof(tmpstr2), "%d", periph->unit_number);
sysctl_ctx_init(&softc->sysctl_ctx);
softc->sysctl_tree = SYSCTL_ADD_NODE(&softc->sysctl_ctx,
SYSCTL_STATIC_CHILDREN(_kern_cam_da), OID_AUTO, tmpstr2,
CTLFLAG_RD, 0, tmpstr);
if (softc->sysctl_tree == NULL) {
printf("daregister: unable to allocate sysctl tree\n");
free(softc, M_DEVBUF);
return (CAM_REQ_CMP_ERR);
}
TASK_INIT(&softc->sysctl_task, 0, dasysctlinit, periph);
/*
* RBC devices don't have to support READ(6), only READ(10).
@ -1049,15 +1077,6 @@ daregister(struct cam_periph *periph, void *arg)
else if (softc->minimum_cmd_size > 12)
softc->minimum_cmd_size = 16;
/*
* Now register the sysctl handler, so the user can the value on
* the fly.
*/
SYSCTL_ADD_PROC(&softc->sysctl_ctx,SYSCTL_CHILDREN(softc->sysctl_tree),
OID_AUTO, "minimum_cmd_size", CTLTYPE_INT | CTLFLAG_RW,
&softc->minimum_cmd_size, 0, dacmdsizesysctl, "I",
"Minimum CDB size");
/*
* Block our timeout handler while we
* add this softc to the dev list.
@ -1539,8 +1558,14 @@ dadone(struct cam_periph *periph, union ccb *done_ccb)
}
}
free(csio->data_ptr, M_TEMP);
if (announce_buf[0] != '\0')
if (announce_buf[0] != '\0') {
xpt_announce_periph(periph, announce_buf);
/*
* Create our sysctl variables, now that we know
* we have successfully attached.
*/
taskqueue_enqueue(taskqueue_thread,&softc->sysctl_task);
}
softc->state = DA_STATE_NORMAL;
/*
* Since our peripheral may be invalidated by an error

View File

@ -36,6 +36,8 @@ __FBSDID("$FreeBSD$");
#include <sys/malloc.h>
#include <sys/mutex.h>
#include <sys/taskqueue.h>
#include <sys/kthread.h>
#include <sys/unistd.h>
static MALLOC_DEFINE(M_TASKQUEUE, "taskqueue", "Task Queues");
@ -44,6 +46,7 @@ static STAILQ_HEAD(taskqueue_list, taskqueue) taskqueue_queues;
static void *taskqueue_ih;
static void *taskqueue_giant_ih;
static struct mtx taskqueue_queues_mutex;
static struct proc *taskqueue_thread_proc;
struct taskqueue {
STAILQ_ENTRY(taskqueue) tq_link;
@ -233,6 +236,31 @@ taskqueue_swi_giant_run(void *dummy)
taskqueue_run(taskqueue_swi_giant);
}
static void
taskqueue_kthread(void *arg)
{
struct mtx kthread_mutex;
bzero(&kthread_mutex, sizeof(kthread_mutex));
mtx_init(&kthread_mutex, "taskqueue kthread", NULL, MTX_DEF);
mtx_lock(&kthread_mutex);
for (;;) {
mtx_unlock(&kthread_mutex);
taskqueue_run(taskqueue_thread);
mtx_lock(&kthread_mutex);
msleep(&taskqueue_thread, &kthread_mutex, PWAIT, "tqthr", 0);
}
}
static void
taskqueue_thread_enqueue(void *context)
{
wakeup(&taskqueue_thread);
}
TASKQUEUE_DEFINE(swi, taskqueue_swi_enqueue, 0,
swi_add(NULL, "task queue", taskqueue_swi_run, NULL, SWI_TQ,
INTR_MPSAFE, &taskqueue_ih));
@ -240,3 +268,7 @@ TASKQUEUE_DEFINE(swi, taskqueue_swi_enqueue, 0,
TASKQUEUE_DEFINE(swi_giant, taskqueue_swi_giant_enqueue, 0,
swi_add(NULL, "Giant task queue", taskqueue_swi_giant_run,
NULL, SWI_TQ_GIANT, 0, &taskqueue_giant_ih));
TASKQUEUE_DEFINE(thread, taskqueue_thread_enqueue, 0,
kthread_create(taskqueue_kthread, NULL,
&taskqueue_thread_proc, RFNOWAIT, 0, "taskqueue"));

View File

@ -107,10 +107,17 @@ SYSINIT(taskqueue_##name, SI_SUB_CONFIGURE, SI_ORDER_SECOND, \
struct __hack
/*
* This queue is serviced by a software interrupt handler. To enqueue
* a task, call taskqueue_enqueue(taskqueue_swi, &task).
* These queues are serviced by software interrupt handlers. To enqueue
* a task, call taskqueue_enqueue(taskqueue_swi, &task) or
* taskqueue_enqueue(taskqueue_swi_giant, &task).
*/
TASKQUEUE_DECLARE(swi_giant);
TASKQUEUE_DECLARE(swi);
/*
* This queue is serviced by a kernel thread. To enqueue a task, call
* taskqueue_enqueue(taskqueue_thread, &task).
*/
TASKQUEUE_DECLARE(thread);
#endif /* !_SYS_TASKQUEUE_H_ */