Use real values to calculate Max I/O size instead of guessing.

Reviewed by:	ken, scottl
Approved by:	ken, scottl, ambrisko (mentors)
MFC after:	3 days
Differential Revision:	https://reviews.freebsd.org/D7043
This commit is contained in:
Stephen McConnell 2016-07-12 19:34:10 +00:00
parent 79156eb7c9
commit 32b0a21e43
8 changed files with 234 additions and 88 deletions

View File

@ -38,7 +38,7 @@
.\" $Id$
.\" $FreeBSD$
.\"
.Dd April 29, 2016
.Dd July 6, 2016
.Dt MPR 4
.Os
.Sh NAME
@ -156,6 +156,29 @@ The current number of active I/O commands is shown in the
dev.mpr.X.io_cmds_active
.Xr sysctl 8
variable.
.Ed
.Pp
To set the maximum number of pages that will be used per I/O for all adapters,
set this tunable in
.Xr loader.conf 5 :
.Bd -literal -offset indent
hw.mpr.max_io_pages=NNNN
.Ed
.Pp
To set the maximum number of pages that will be used per I/O for a specific
adapter, set this tunable in
.Xr loader.conf 5 :
.Bd -literal -offset indent
dev.mpr.X.max_io_pages=NNNN
.Ed
.Pp
The default max_io_pages value is -1, meaning that the maximum I/O size that
will be used per I/O will be calculated using the IOCFacts values stored in
the controller.
The lowest value that the driver will use for max_io_pages is 1, otherwise
IOCFacts will be used to calculate the maximum I/O size.
The smaller I/O size calculated from either max_io_pages or IOCFacts will be the
maximum I/O size used by the driver.
.Pp
The highest number of active I/O commands seen since boot is stored in the
dev.mpr.X.io_cmds_highwater
@ -220,7 +243,7 @@ SATA disks that take several seconds to spin up and fail the SATA Identify
command might not be discovered by the driver.
This problem can sometimes be overcome by increasing the value of the spinup
wait time in
.Xr loader.conf 5 :
.Xr loader.conf 5
with the
.Bd -literal -offset indent
hw.mpr.spinup_wait_time=NNNN

View File

@ -1,5 +1,8 @@
.\"
.\" Copyright (c) 2010 Spectra Logic Corporation
.\" Copyright (c) 2014 LSI Corp
.\" Copyright (c) 2016 Avago Technologies
.\" Copyright (c) 2016 Broadcom Ltd.
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
@ -30,25 +33,27 @@
.\" mps driver man page.
.\"
.\" Author: Ken Merry <ken@FreeBSD.org>
.\" Author: Stephen McConnell <slm@FreeBSD.org>
.\"
.\" $Id: //depot/SpectraBSD/head/share/man/man4/mps.4#6 $
.\" $FreeBSD$
.\"
.Dd December 9, 2015
.Dd July 5, 2016
.Dt MPS 4
.Os
.Sh NAME
.Nm mps
.Nd LSI Fusion-MPT 2 Serial Attached SCSI driver
.Nd "LSI Fusion-MPT 2 IT/IR 6Gb/s Serial Attached SCSI/SATA driver"
.Sh SYNOPSIS
To compile this driver into your kernel,
place the following lines in your kernel configuration file:
To compile this driver into the kernel, place these lines in the kernel
configuration file:
.Bd -ragged -offset indent
.Cd "device pci"
.Cd "device scbus"
.Cd "device mps"
.Ed
.Pp
Or, to load the driver as a module at boot, place the following line in
The driver can be loaded as a module at boot time by placing this line in
.Xr loader.conf 5 :
.Bd -literal -offset indent
mps_load="YES"
@ -56,35 +61,30 @@ mps_load="YES"
.Sh DESCRIPTION
The
.Nm
driver provides support for LSI Logic Fusion-MPT 2
driver provides support for Broadcom Ltd./Avago Tech (LSI)
Fusion-MPT 2 IT/IR
.Tn SAS
controllers and WarpDrive solid state storage cards.
.Sh HARDWARE
The
These controllers are supported by the
.Nm
driver supports the following hardware:
driver:
.Pp
.Bl -bullet -compact
.It
LSI Logic SAS2004 (4 Port
.Tn SAS )
Broadcom Ltd./Avago Tech (LSI) SAS 2004 (4 Port SAS)
.It
LSI Logic SAS2008 (8 Port
.Tn SAS )
Broadcom Ltd./Avago Tech (LSI) SAS 2008 (8 Port SAS)
.It
LSI Logic SAS2108 (8 Port
.Tn SAS )
Broadcom Ltd./Avago Tech (LSI) SAS 2108 (8 Port SAS)
.It
LSI Logic SAS2116 (16 Port
.Tn SAS )
Broadcom Ltd./Avago Tech (LSI) SAS 2116 (16 Port SAS)
.It
LSI Logic SAS2208 (8 Port
.Tn SAS )
Broadcom Ltd./Avago Tech (LSI) SAS 2208 (8 Port SAS)
.It
LSI Logic SAS2308 (8 Port
.Tn SAS )
Broadcom Ltd./Avago Tech (LSI) SAS 2308 (8 Port SAS)
.It
LSI Logic SSS6200 Solid State Storage
Broadcom Ltd./Avago Tech (LSI) SSS6200 Solid State Storage
.It
Intel Integrated RAID Module RMS25JB040
.It
@ -95,9 +95,12 @@ Intel Integrated RAID Module RMS25KB040
Intel Integrated RAID Module RMS25KB080
.El
.Sh CONFIGURATION
.Pp
In all tunable descriptions below, X represents the adapter number.
.Pp
To disable MSI interrupts for all
.Nm
driver instances, set the following tunable value in
driver instances, set this tunable value in
.Xr loader.conf 5 :
.Bd -literal -offset indent
hw.mps.disable_msi=1
@ -105,17 +108,15 @@ hw.mps.disable_msi=1
.Pp
To disable MSI interrupts for a specific
.Nm
driver instance, set the following tunable value in
driver instance, set this tunable value in
.Xr loader.conf 5 :
.Bd -literal -offset indent
dev.mps.X.disable_msi=1
.Ed
.Pp
where X is the adapter number.
.Pp
To disable MSI-X interrupts for all
.Nm
driver instances, set the following tunable value in
driver instances, set this tunable value in
.Xr loader.conf 5 :
.Bd -literal -offset indent
hw.mps.disable_msix=1
@ -123,84 +124,157 @@ hw.mps.disable_msix=1
.Pp
To disable MSI-X interrupts for a specific
.Nm
driver instance, set the following tunable value in
driver instance, set this tunable value in
.Xr loader.conf 5 :
.Bd -literal -offset indent
dev.mps.X.disable_msix=1
.Ed
.Pp
where X is the adapter number.
.Pp
To set the maximum number of DMA chains allocated for all adapters,
set the following variable in
To set the maximum number of DMA chains allocated for all adapters, set this
tunable in
.Xr loader.conf 5 :
.Bd -literal -offset indent
hw.mps.max_chains=NNNN
.Ed
.Pp
To set the maximum number of DMA chains allocated for a specific adapter,
set the following variable in
set this tunable in
.Xr loader.conf 5 :
.Bd -literal -offset indent
dev.mps.X.max_chains=NNNN
.Ed
.Pp
This variable may also be viewed via
.Xr sysctl 8
to see the maximum set for a given adapter.
The default max_chains value is 2048.
.Pp
The current number of free chain frames may be seen via the
The current number of free chain frames is stored in the
dev.mps.X.chain_free
.Xr sysctl 8
variable.
.Pp
The lowest number of free chain frames may be seen via the
The lowest number of free chain frames seen since boot is stored in the
dev.mps.X.chain_free_lowwater
.Xr sysctl 8
variable.
.Pp
The number of times that chain frame allocations have failed since boot is
stored in the
dev.mps.X.chain_alloc_fail
.Xr sysctl 8
variable.
This can be used to determine whether the max_chains tunable should be
increased to help performance.
.Pp
The current number of active I/O commands is shown in the
dev.mps.X.io_cmds_active
.Xr sysctl 8
variable.
.Ed
.Pp
The maximum number of active I/O command seen since boot is shown in the
To set the maximum number of pages that will be used per I/O for all adapters,
set this tunable in
.Xr loader.conf 5 :
.Bd -literal -offset indent
hw.mps.max_io_pages=NNNN
.Ed
.Pp
To set the maximum number of pages that will be used per I/O for a specific
adapter, set this tunable in
.Xr loader.conf 5 :
.Bd -literal -offset indent
dev.mps.X.max_io_pages=NNNN
.Ed
.Pp
The default max_io_pages value is -1, meaning that the maximum I/O size that
will be used per I/O will be calculated using the IOCFacts values stored in
the controller.
The lowest value that the driver will use for max_io_pages is 1, otherwise
IOCFacts will be used to calculate the maximum I/O size.
The smaller I/O size calculated from either max_io_pages or IOCFacts will be the
maximum I/O size used by the driver.
.Pp
The highest number of active I/O commands seen since boot is stored in the
dev.mps.X.io_cmds_highwater
.Xr sysctl 8
variable.
.Pp
Devices can be excluded from
.Nm
control for all adapters by setting this tunable in
.Xr loader.conf 5 :
.Bd -literal -offset indent
hw.mps.exclude_ids=Y
.Ed
.Pp
Y represents the target ID of the device.
If more than one device is to be excluded, target IDs are separated by commas.
.Pp
Devices can be excluded from
.Nm
control for a specific adapter by setting this tunable in
.Xr loader.conf 5 :
.Bd -literal -offset indent
dev.mps.X.exclude_ids=Y
.Ed
.Pp
Y represents the target ID of the device.
If more than one device is to be excluded, target IDs are separated by commas.
.Pp
The adapter can issue the
.Sy StartStopUnit
SCSI command to SATA direct-access devices during shutdown, to allow the
device to quiesce before being powered down.
SCSI command to SATA direct-access devices during shutdown.
This allows the device to quiesce powering down.
To control this feature for all adapters, set the
.Bd -literal -offset indent
hw.mps.enable_ssu
.Ed
.Pp
tunable value in
tunable in
.Xr loader.conf 5
to one of the following values:
to one of these values:
.Bl -tag -width 6n -offset indent
.It 0
Do not send SSU to either HDDs or SSDs.
.It 1
Send SSU to SSDs, but not to HDDs; this is the default value.
Send SSU to SSDs, but not to HDDs.
This is the default value.
.It 2
Send SSU to HDDs, but not to SSDs.
.It 3
Send SSU to both HDDs and SSDs.
.El
.Pp
To control the feature for a specific adapter, set the following tunable
value in
To control the feature for a specific adapter, set this tunable value in
.Xr loader.conf 5 :
.Bd -literal -offset indent
dev.mps.X.enable_ssu
.Ed
.Pp
where X is the adapter number.
The same set of values are valid as for all adapters.
The same set of values are valid when setting this tunable for all adapters.
.Pp
SATA disks that take several seconds to spin up and fail the SATA Identify
command might not be discovered by the driver.
This problem can sometimes be overcome by increasing the value of the spinup
wait time in
.Xr loader.conf 5
with the
.Bd -literal -offset indent
hw.mps.spinup_wait_time=NNNN
.Ed
.Pp
tunable.
NNNN represents the number of seconds to wait for SATA devices to spin up when
the device fails the initial SATA Identify command.
.Pp
Spinup wait times can be set for specific adapters in
.Xr loader.conf 5 :
with the
.Bd -literal -offset indent
dev.mps.X.spinup_wait_time=NNNN
.Ed
.Pp
tunable.
NNNN is the number of seconds to wait for SATA devices to spin up when they fail
the initial SATA Identify command.
.Sh DEBUGGING
To enable debugging prints from the
.Nm
@ -209,25 +283,30 @@ driver, set the
hw.mps.X.debug_level
.Ed
.Pp
variable, where X is the adapter number, either in
tunable, either in
.Xr loader.conf 5
or via
or by using
.Xr sysctl 8 .
The following bits have the described effects:
.Bl -tag -width 6n -offset indent
.It 0x01
Enable informational prints.
.It 0x02
Enable tracing prints.
.It 0x04
Enable prints for driver faults.
.It 0x08
Enable prints for controller events.
.El
These bits have the described effects:
.Bd -literal -offset indent
0x0001 Enable informational prints (set by default).
0x0002 Enable prints for driver faults (set by default).
0x0004 Enable prints for controller events.
0x0008 Enable prints for controller logging.
0x0010 Enable prints for tracing recovery operations.
0x0020 Enable prints for parameter errors and programming bugs.
0x0040 Enable prints for system initialization operations.
0x0080 Enable prints for more detailed information.
0x0100 Enable prints for user-generated commands (IOCTL).
0x0200 Enable prints for device mapping.
0x0400 Enable prints for tracing through driver functions.
.Ed
.Sh SEE ALSO
.Xr cam 4 ,
.Xr cd 4 ,
.Xr ch 4 ,
.Xr da 4 ,
.Xr mpr 4 ,
.Xr mpt 4 ,
.Xr pci 4 ,
.Xr sa 4 ,
@ -238,24 +317,17 @@ Enable prints for controller events.
.Sh HISTORY
The
.Nm
driver first appeared in
.Fx 9.0 .
driver first appeared in FreeBSD 9.3.
.Sh AUTHORS
.An -nosplit
The
.Nm
driver was originally written by
.An -nosplit
.An Scott Long Aq Mt scottl@FreeBSD.org .
It has been improved and tested by LSI Logic Corporation.
It has been improved and tested by LSI Corporation,
Avago Technologies (formally LSI), and Broadcom Ltd. (formally Avago).
.Pp
This man page was written by
.An Ken Merry Aq Mt ken@FreeBSD.org .
.Sh BUGS
This driver has a couple of known shortcomings:
.Bl -bullet -compact
.It
No userland utility available (e.g.,
.Xr mptutil 8 ) .
.It
The driver probes devices sequentially.
If your system has a large number of devices, the probe will take a while.
.El
.An Ken Merry Aq Mt ken@FreeBSD.org
with additional input from
.An Stephen McConnell Aq Mt slm@FreeBSD.org .

View File

@ -1373,6 +1373,7 @@ mpr_get_tunables(struct mpr_softc *sc)
sc->disable_msix = 0;
sc->disable_msi = 0;
sc->max_chains = MPR_CHAIN_FRAMES;
sc->max_io_pages = MPR_MAXIO_PAGES;
sc->enable_ssu = MPR_SSU_ENABLE_SSD_DISABLE_HDD;
sc->spinup_wait_time = DEFAULT_SPINUP_WAIT;
@ -1383,6 +1384,7 @@ mpr_get_tunables(struct mpr_softc *sc)
TUNABLE_INT_FETCH("hw.mpr.disable_msix", &sc->disable_msix);
TUNABLE_INT_FETCH("hw.mpr.disable_msi", &sc->disable_msi);
TUNABLE_INT_FETCH("hw.mpr.max_chains", &sc->max_chains);
TUNABLE_INT_FETCH("hw.mpr.max_io_pages", &sc->max_io_pages);
TUNABLE_INT_FETCH("hw.mpr.enable_ssu", &sc->enable_ssu);
TUNABLE_INT_FETCH("hw.mpr.spinup_wait_time", &sc->spinup_wait_time);
@ -1403,6 +1405,10 @@ mpr_get_tunables(struct mpr_softc *sc)
device_get_unit(sc->mpr_dev));
TUNABLE_INT_FETCH(tmpstr, &sc->max_chains);
snprintf(tmpstr, sizeof(tmpstr), "dev.mpr.%d.max_io_pages",
device_get_unit(sc->mpr_dev));
TUNABLE_INT_FETCH(tmpstr, &sc->max_io_pages);
bzero(sc->exclude_ids, sizeof(sc->exclude_ids));
snprintf(tmpstr, sizeof(tmpstr), "dev.mpr.%d.exclude_ids",
device_get_unit(sc->mpr_dev));
@ -1487,6 +1493,11 @@ mpr_setup_sysctl(struct mpr_softc *sc)
OID_AUTO, "max_chains", CTLFLAG_RD,
&sc->max_chains, 0,"maximum chain frames that will be allocated");
SYSCTL_ADD_INT(sysctl_ctx, SYSCTL_CHILDREN(sysctl_tree),
OID_AUTO, "max_io_pages", CTLFLAG_RD,
&sc->max_io_pages, 0,"maximum pages to allow per I/O (if <1 use "
"IOCFacts)");
SYSCTL_ADD_INT(sysctl_ctx, SYSCTL_CHILDREN(sysctl_tree),
OID_AUTO, "enable_ssu", CTLFLAG_RW, &sc->enable_ssu, 0,
"enable SSU to SATA SSD/HDD at shutdown");

View File

@ -971,6 +971,8 @@ mprsas_action(struct cam_sim *sim, union ccb *ccb)
case XPT_PATH_INQ:
{
struct ccb_pathinq *cpi = &ccb->cpi;
struct mpr_softc *sc = sassc->sc;
uint8_t sges_per_frame;
cpi->version_num = 1;
cpi->hba_inquiry = PI_SDTR_ABLE|PI_TAG_ABLE|PI_WIDE_16;
@ -999,13 +1001,23 @@ mprsas_action(struct cam_sim *sim, union ccb *ccb)
cpi->transport_version = 0;
cpi->protocol = PROTO_SCSI;
cpi->protocol_version = SCSI_REV_SPC;
#if __FreeBSD_version >= 800001
/*
* XXXSLM-probably need to base this number on max SGL's and
* page size.
* Max IO Size is Page Size * the following:
* ((SGEs per frame - 1 for chain element) *
* Max Chain Depth) + 1 for no chain needed in last frame
*
* If user suggests a Max IO size to use, use the smaller of the
* user's value and the calculated value as long as the user's
* value is larger than 0. The user's value is in pages.
*/
cpi->maxio = 256 * 1024;
#endif
sges_per_frame = (sc->chain_frame_size /
sizeof(MPI2_IEEE_SGE_SIMPLE64)) - 1;
cpi->maxio = (sges_per_frame * sc->facts->MaxChainDepth) + 1;
cpi->maxio *= PAGE_SIZE;
if ((sc->max_io_pages > 0) && (sc->max_io_pages * PAGE_SIZE <
cpi->maxio))
cpi->maxio = sc->max_io_pages * PAGE_SIZE;
mprsas_set_ccbstatus(ccb, CAM_REQ_CMP);
break;
}

View File

@ -33,7 +33,7 @@
#ifndef _MPRVAR_H
#define _MPRVAR_H
#define MPR_DRIVER_VERSION "13.00.00.00-fbsd"
#define MPR_DRIVER_VERSION "13.01.00.00-fbsd"
#define MPR_DB_MAX_WAIT 2500
@ -41,6 +41,7 @@
#define MPR_EVT_REPLY_FRAMES 32
#define MPR_REPLY_FRAMES MPR_REQ_FRAMES
#define MPR_CHAIN_FRAMES 2048
#define MPR_MAXIO_PAGES (-1)
#define MPR_SENSE_LEN SSD_FULL_SIZE
#define MPR_MSI_COUNT 1
#define MPR_SGE64_SIZE 12
@ -264,6 +265,7 @@ struct mpr_softc {
int io_cmds_highwater;
int chain_free;
int max_chains;
int max_io_pages;
int chain_free_lowwater;
uint32_t chain_frame_size;
uint16_t chain_seg_size;

View File

@ -1350,6 +1350,7 @@ mps_get_tunables(struct mps_softc *sc)
sc->disable_msix = 0;
sc->disable_msi = 0;
sc->max_chains = MPS_CHAIN_FRAMES;
sc->max_io_pages = MPS_MAXIO_PAGES;
sc->enable_ssu = MPS_SSU_ENABLE_SSD_DISABLE_HDD;
sc->spinup_wait_time = DEFAULT_SPINUP_WAIT;
@ -1360,6 +1361,7 @@ mps_get_tunables(struct mps_softc *sc)
TUNABLE_INT_FETCH("hw.mps.disable_msix", &sc->disable_msix);
TUNABLE_INT_FETCH("hw.mps.disable_msi", &sc->disable_msi);
TUNABLE_INT_FETCH("hw.mps.max_chains", &sc->max_chains);
TUNABLE_INT_FETCH("hw.mps.max_io_pages", &sc->max_io_pages);
TUNABLE_INT_FETCH("hw.mps.enable_ssu", &sc->enable_ssu);
TUNABLE_INT_FETCH("hw.mps.spinup_wait_time", &sc->spinup_wait_time);
@ -1380,6 +1382,10 @@ mps_get_tunables(struct mps_softc *sc)
device_get_unit(sc->mps_dev));
TUNABLE_INT_FETCH(tmpstr, &sc->max_chains);
snprintf(tmpstr, sizeof(tmpstr), "dev.mps.%d.max_io_pages",
device_get_unit(sc->mps_dev));
TUNABLE_INT_FETCH(tmpstr, &sc->max_io_pages);
bzero(sc->exclude_ids, sizeof(sc->exclude_ids));
snprintf(tmpstr, sizeof(tmpstr), "dev.mps.%d.exclude_ids",
device_get_unit(sc->mps_dev));
@ -1464,6 +1470,11 @@ mps_setup_sysctl(struct mps_softc *sc)
OID_AUTO, "max_chains", CTLFLAG_RD,
&sc->max_chains, 0,"maximum chain frames that will be allocated");
SYSCTL_ADD_INT(sysctl_ctx, SYSCTL_CHILDREN(sysctl_tree),
OID_AUTO, "max_io_pages", CTLFLAG_RD,
&sc->max_io_pages, 0,"maximum pages to allow per I/O (if <1 use "
"IOCFacts)");
SYSCTL_ADD_INT(sysctl_ctx, SYSCTL_CHILDREN(sysctl_tree),
OID_AUTO, "enable_ssu", CTLFLAG_RW, &sc->enable_ssu, 0,
"enable SSU to SATA SSD/HDD at shutdown");

View File

@ -928,6 +928,8 @@ mpssas_action(struct cam_sim *sim, union ccb *ccb)
case XPT_PATH_INQ:
{
struct ccb_pathinq *cpi = &ccb->cpi;
struct mps_softc *sc = sassc->sc;
uint8_t sges_per_frame;
cpi->version_num = 1;
cpi->hba_inquiry = PI_SDTR_ABLE|PI_TAG_ABLE|PI_WIDE_16;
@ -951,12 +953,23 @@ mpssas_action(struct cam_sim *sim, union ccb *ccb)
cpi->transport_version = 0;
cpi->protocol = PROTO_SCSI;
cpi->protocol_version = SCSI_REV_SPC;
#if __FreeBSD_version >= 800001
/*
* XXX KDM where does this number come from?
* Max IO Size is Page Size * the following:
* ((SGEs per frame - 1 for chain element) *
* Max Chain Depth) + 1 for no chain needed in last frame
*
* If user suggests a Max IO size to use, use the smaller of the
* user's value and the calculated value as long as the user's
* value is larger than 0. The user's value is in pages.
*/
cpi->maxio = 256 * 1024;
#endif
sges_per_frame = ((sc->facts->IOCRequestFrameSize * 4) /
sizeof(MPI2_SGE_SIMPLE64)) - 1;
cpi->maxio = (sges_per_frame * sc->facts->MaxChainDepth) + 1;
cpi->maxio *= PAGE_SIZE;
if ((sc->max_io_pages > 0) && (sc->max_io_pages * PAGE_SIZE <
cpi->maxio))
cpi->maxio = sc->max_io_pages * PAGE_SIZE;
mpssas_set_ccbstatus(ccb, CAM_REQ_CMP);
break;
}

View File

@ -33,7 +33,7 @@
#ifndef _MPSVAR_H
#define _MPSVAR_H
#define MPS_DRIVER_VERSION "21.00.00.00-fbsd"
#define MPS_DRIVER_VERSION "21.01.00.00-fbsd"
#define MPS_DB_MAX_WAIT 2500
@ -41,6 +41,7 @@
#define MPS_EVT_REPLY_FRAMES 32
#define MPS_REPLY_FRAMES MPS_REQ_FRAMES
#define MPS_CHAIN_FRAMES 2048
#define MPS_MAXIO_PAGES (-1)
#define MPS_SENSE_LEN SSD_FULL_SIZE
#define MPS_MSI_COUNT 1
#define MPS_SGE64_SIZE 12
@ -280,6 +281,7 @@ struct mps_softc {
int io_cmds_highwater;
int chain_free;
int max_chains;
int max_io_pages;
int chain_free_lowwater;
u_int enable_ssu;
int spinup_wait_time;