Commit Graph

43 Commits

Author SHA1 Message Date
Steven Hartland
7d147b81ea Fix mps deadlock when handling panic
During shutdown mps waits for its SSU requests to complete however when
performing a reboot after handling a panic the scheduler is stopped so
getmicrotime which is used can be non-functional.

Switch to using the same method as shutdown_panic to ensure we actually
complete.

In addition reduce the timeout when RB_NOSYNC is set in howto as we expect
this to fail.

Reviewed by:	slm
MFC after:	1 week
Sponsored by:	Multiplay
Differential Revision:	https://reviews.freebsd.org/D12776
2018-03-14 21:32:23 +00:00
Alexander Motin
731308d01d Allow physically non-contiguous chain frames allocation in mps(4)/mpr(4).
Chain frames required to satisfy all 2K of declared I/Os of 128KB each take
more then a megabyte of a physical memory, all of which existing code tries
allocate as physically contiguous.  This patch removes that physical
contiguousness requirement, leaving only virtual contiguousness.  I was
thinking about other ways of allocation, but the less granular allocation
becomes, the bigger is the overhead and/or complexity, reaching about 100%
overhead if allocate each frame separately.

The patch also bumps the chain frames hard limit from 2K to 16K.  It is more
than enough for the case of default REQ_FRAMES and MAXPHYS (the drivers will
allocate less than that automatically), while in case of increased MAXPHYS
it will control maximal memory usage.

Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D14420
2018-02-27 01:48:13 +00:00
Scott Long
f0779b0452 Improve command lifecycle debugging and detection of problems.
Sponsored by:	Netflix
2018-02-18 16:41:34 +00:00
Alexander Motin
4f5d657343 Teach mps(4) and mpr(4) drivers to autotune chain frames.
This is a first part of the change.  It makes the drivers to calculate
the required number of chain frames to satisfy worst case scenarios, but
it does not change existing overly strict limits on them.  The next step
will be to rewrite the allocator to not require megabytes of physically
contiguous address space, that may be problematic if done after boot,
after doing which the limits can be removed.  Until that this code can
just correct user set limits, if they are set too high.

Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D14261
2018-02-10 00:55:46 +00:00
Scott Long
964107031b Cache the value of the request and reply frame size since it's used quite
a bit in the normal operation of the driver.  Covert it to represent bytes
instead of 32bit words.  Fix what I believe to be is a bug in this respect
with the Tri-mode cards.

Sponsored by:	Netflix
2018-02-06 21:01:38 +00:00
Alexander Motin
62a09ee976 Fix queue length reporting in mps(4) and mpr(4).
Both drivers were found to report CAM bigger queue depth then they really
can handle.  It made them later under high load with many disks return
some of submitted requests back with CAM_REQUEUE_REQ status for later
resubmission.

Reviewed by:	scottl
MFC after:	1 week
Sponsored by:	iXsystems, Inc.
Differential Revision:	https://reviews.freebsd.org/D14215
2018-02-06 16:02:25 +00:00
Kenneth D. Merry
e2997a03b7 Diagnostic buffer fixes for the mps(4) and mpr(4) drivers.
In mp{r,s}_diag_register(), which is used to register diagnostic
buffers with the mp{r,s}(4) firmware, we allocate DMAable memory.

There were several issues here:
 o No checking of the bus_dmamap_load() return value.  If the load
   failed or got deferred, mp{r,s}_diag_register() continued on as if
   nothing had happened.  We now check the return value and bail
   out if it fails.

 o No waiting for a deferred load callback.  bus_dmamap_load()
   calls a supplied callback when the mapping is done.  This is
   generally done immediately, but it can be deferred.
   mp{r,s}_diag_register() did not check to see whether the callback
   was already done before proceeding on.  We now sleep until the
   callback is done if it is deferred.

 o No call to bus_dmamap_sync(... BUS_DMASYNC_PREREAD) after the
   memory is allocated and loaded.  This is necessary on some
   platforms to synchronize host memory that is going to be updated
   by a device.

Both drivers would also panic if the firmware was reinitialized while
a diagnostic buffer operation was in progress.  This fixes that problem
as well.  (The driver will reinitialize the firmware in various
circumstances, but the problem I ran into was that the firmware would
generate an IOC Fault due to a PCIe error.)

mp{r,s}var.h:
	Add a new structure, struct mpr_busdma_context, that is
	used for deferred busdma load callbacks.

	Add a prototype for mp{r,s}_memaddr_wait_cb().
mp{r,s}.c:
	Add a new busdma callback function, mp{r,s}_memaddr_wait_cb().
	This provides synchronization for callers that want to
	wait on a deferred bus_dmamap_load() callback.

mp{r,s}_user.c:
	In bus_dmamap_register(), add a call to bus_dmamap_sync()
	with the BUS_DMASYNC_PREREAD flag set after an allocation
	is loaded.

	Also, check the return value of bus_dmamap_load().  If it
	fails, bail out.  If it is EINPROGRESS, wait for the
	callback to happen.  We use an interruptible sleep (msleep
	with PCATCH) and let the callback clean things up if we get
	interrupted.

	In mpr_diag_read_buffer() and mps_diag_read_buffer(), call
	bus_dmamap_sync(..., BUS_DMASYNC_POSTREAD) before copying
	the data out to make sure the data is in stable storage.

	In mp{r,s}_post_fw_diag_buffer() and
	mp{r,s}_release_fw_diag_buffer(), check the reply to see
	whether it is NULL.  It can be NULL (and the command non-NULL)
	if the controller gets reinitialized while we're waiting for
	the command to complete but the driver structures aren't
	reallocated.  The driver structures generally won't be
	reallocated unless there is a firmware upgrade that changes
	one of the IOCFacts.

	When freeing diagnostic buffers in mp{r,s}_diag_register()
	and mp{r,s}_diag_unregister(), zero/NULL out the buffer after
	freeing it.  This will prevent a duplicate free in some
	situations.

Sponsored by:	Spectra Logic
Reviewed by:	mav, scottl
MFC after:	1 week
Differential Revision:	D13453
2018-02-06 15:58:22 +00:00
Pedro F. Giffuni
718cf2ccb9 sys/dev: further adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
2017-11-27 14:52:40 +00:00
Scott Long
3c5ac992c7 Add infrastructure for allocating multiple MSI-X interrupts. Also
add more fine-tuned controls for allocating requests and replies.

Sponsored by:	Netflix
2017-09-11 01:51:27 +00:00
Scott Long
a7d065b3af Remove the unnecessary use of a temporary string buffer.
Sponsored by:	Netflix
2017-09-09 18:39:55 +00:00
Scott Long
bec09074ca Start separating the LSI drivers into per-queue structures. No
functional change.

Sponsored by:	Netflix
2017-09-09 18:03:40 +00:00
Kenneth D. Merry
6d4ffcb4ac Changes to make mps(4) and mpr(4) handle reinit with reallocation.
When the mps(4) and mpr(4) drivers need to reinitialize the
firmware, they sometimes need to reallocate all of the memory
allocated by the driver.  The reallocation happens whenever the IOC
Facts change.  That should only happen after a firmware upgrade.

If the reinitialization happens as a result of a timed out command
sent to the card, the command that timed out and triggered the
reinit may have been freed if iocfacts_allocate() reallocated all
memory.  If the caller attempts to access the command after that,
the kernel will panic because the caller will be dereferencing
freed memory.

The solution is to set a flag in the softc when we reallocate,
and avoid dereferencing the command strucure if we've reallocated.

The changes are largely the same in both drivers, since mpr(4) is a
derivative of mps(4).

 o In iocfacts_allocate(), if the IOC Facts have changed and we
   need to reallocate, set the REALLOCATED flag in the softc.

 o Change wait_command() to take a struct mps_command ** instead of
   a struct mps_command *.  This allows us to NULL out the caller's
   command pointer if we have to reinit the controller and the data
   structures get reallocated.  (The REALLOCATED flag will be set
   in the softc if that has happened.)

 o In every place that calls wait_command(), make sure we handle
   the case where the command is NULL after the call.

 o The mpr(4) driver has mpr_request_polled() which can also
   reinitialize the card.  Also check for reallocation there.

Reviewed by:	scottl, slm
MFC after:	1 week
Sponsored by:	Spectra Logic
2017-08-10 14:59:17 +00:00
Scott Long
252b2b4f4f Split the interrupt setup code into two parts: allocation and configuration.
Do the allocation before requesting the IOCFacts message.  This triggers
    the LSI firmware to recognize the multiqueue should be enabled if available.
    Multiqueue isn't used by the driver yet, but this also fixes a problem with
    the cached IOCFacts not matching latter checks, leading to potential problems
    with error recovery.

    As a side-effect, fetch the driver tunables as early as possible.

Reviewed by:	slm
Obtained from:	Netflix
Differential Revision:	D9243
2017-07-30 06:53:58 +00:00
Stephen McConnell
635e58c715 Fix several problems with mapping code.
Reviewed by:    ken, scottl, asomers, ambrisko, mav
Approved by:	ken, mav
MFC after:      1 week
Differential Revision: https://reviews.freebsd.org/D10878
2017-05-25 19:14:44 +00:00
Scott Long
c11c484f15 Rework the debug print API. Event printing no longer gets special handling.
All of the printing from the tables file now has wrappers so that the
handling is cleaner and it's possible to print something out (say, during
development) without having to fight the global debug flags. This re-org
will also make it easier to have the tables be compiled out at build time
if desired.

Other than fixing some minor bugs, there are no user-visible changes from
this change

Sponsored by:	Netflix, Inc.
Differential Revision:	D9238
2017-01-19 21:47:50 +00:00
Scott Long
4ab1cdc5ad Add a fallback to the device mapper logic. We've seen systems in the field
that are apparently misconfigured by the manufacturer and cause the mapping
logic to fail.  The fallback allows drive numbers to be assigned based on the
PHY number that they're attached to.  Add sysctls and tunables to overrid
this new behavior, but they should be considered only necessary for debugging.

Reviewed by:	 imp, smh
Obtained from:	Netflix
MFC after:	3 days
Sponsored by:	D8403
2016-11-02 15:13:25 +00:00
Stephen McConnell
32b0a21e43 Use real values to calculate Max I/O size instead of guessing.
Reviewed by:	ken, scottl
Approved by:	ken, scottl, ambrisko (mentors)
MFC after:	3 days
Differential Revision:	https://reviews.freebsd.org/D7043
2016-07-12 19:34:10 +00:00
Stephen McConnell
f4e69c98fd - No log bit in IOCStatus and endian-safe changes.
Use MPI2_IOCSTATUS_MASK when checking IOCStatus to mask off the log bit, and
make a few more things endian-safe.

- Fix possible use of invalid pointer.

It was possible to use an invalid pointer to get the target ID value. To fix
this, initialize a local Target ID variable to an invalid value and change that
variable to a valid value only if the pointer to the Target ID is not NULL.

- No need to set the MPSSAS_SHUTDOWN flag because it's never used.

- done_ccb pointer can be used if it is NULL.

To prevent this, move check for done_ccb == NULL to before done_ccb is used in
mpssas_stop_unit_done().

- Disks can go missing until a reboot is done in some cases.

This is due to the DevHandle not being released, which causes the Firmware to
not allow that disk to be re-added.

Reviewed by:	ken
Approved by:	re (gjb), ken, scottl, ambrisko (mentors)
MFC after:	1 week
Differential Revision:	https://reviews.freebsd.org/D6872
2016-06-20 18:14:51 +00:00
Scott Long
ee5c196b0a Add sysctls for dumping out the device mapping tables. I'm finding this
useful for debugging device-target translation bugs.

MFC after:	3 days
Sponsored by:	Netflix
2016-02-04 23:38:55 +00:00
Scott Long
d0be3479d5 Remove _FreeBSD_version check for something that was only an issue with
9-CURRENT.

Obtained from:  Netlfix, Inc
MFC after:      3 days
2015-10-16 17:56:43 +00:00
Scott Long
ac7d1ed24c Remove mps_request_polled(), it's redundant to mps_wait_command()
Obtained from:	Netflix, Inc.
2015-08-10 09:02:34 +00:00
Scott Long
44f2b87692 Remove some unused code.
Obtained from:	Netflix, Inx.
2015-08-10 08:57:34 +00:00
Stephen McConnell
ef065d89e9 - Updated all files with 2015 Avago copyright, and updated LSI's copyright
dates.

- Changed all of the PCI device strings from LSI to Avago Technologies (LSI).

- Added a sysctl variable to control how StartStopUnit behavior works. User can
  select to spin down disks based on if disk is SSD or HDD.

- Inquiry data is required to tell if a disk will support SSU at shutdown or
  not. Due to the addition of mpssas_async, which gets Advanced Info but not
  Inquiry data, the setting of supports_SSU was moved to the
  mpssas_scsiio_complete function, which snoops for any Inquiry commands. And,
  since disks are shutdown as a target and not a LUN, this process was
  simplified by basing it on targets and not LUNs.

- Added a sysctl variable that sets the amount of time to retry after sending a
  failed SATA ID command. This helps with some bad disks and large disks that
  require a lot of time to spin up. Part of this change was to add a callout to
  handle timeouts with the SATA ID command. The callout function is called
  mpssas_ata_id_timeout(). (Fixes PR 191348)

- Changed the way resets work by allowing I/O to continue to devices that are
  not currently under a reset condition. This uses devq's instead of simq's and
  makes use of the MPSSAS_TARGET_INRESET flag. This change also adds a function
  called mpssas_prepare_tm().

- Some changes were made to reduce code duplication when getting a SAS address
  for a SATA disk.

- Fixed some formatting and whitespace.

- Bump version of mps driver to 20.00.00.00-fbsd

PR:		191348
Reviewed by:	ken, scottl
Approved by:	ken, scottl
MFC after:	2 weeks
2015-02-24 22:07:42 +00:00
Steven Hartland
e248a3d1f6 Bring in LSI's phase19 changes
* Removed unused mpssas_discovery_timeout function.
* Don't alter mapping boundaries if not raid firmware.
* Check free_busaddr instead of post_busaddr (diff minimisation really)

MFC after:	2 weeks
2014-07-30 18:21:06 +00:00
Steven Hartland
7571e7f64a Bring in LSI's phase16 - phase18 changes
* Implements Start Stop Unit for SATA direct-attach devices in IR mode to avoid
  data corruption.
* Use CAM_DEV_NOT_THERE instead of CAM_SEL_TIMEOUT and CAM_TID_INVALID

Obtained from:	LSI
MFC after:	2 weeks
2014-07-30 17:58:17 +00:00
Scott Long
b7f7712702 Refactor some code in mps.c to reduce header pollution.
Reviewed by:	gibbs
Obtained from:	Netflix, Inc.
MFC after:	2 days
2014-07-01 04:33:36 +00:00
Scott Long
fe8391035a Rate limit the 'out of chain frame' messages to once per 60 seconds.
Obtained from:	Netflix
MFC after:	3 days
2013-08-09 01:10:33 +00:00
Scott Long
d9802deb4e Sometimes a device misbehaves so badly that it disrupts the entire system.
Add a tunable that allows such a device to be excluded from the driver.
The id parameter is the target id that the driver assigns to a given device.

dev.mps.X.exclude_ids=<id>,<id>

Obtained from:	Netflix
MFC after:	3 days
2013-08-09 01:09:02 +00:00
Kenneth D. Merry
9b91b192fb Merge in phase 14+ -> 16 mps driver fixes from LSI:
---------------------------------------------------------------
System panics during a Port reset with ouststanding I/O
---------------------------------------------------------------
It is possible to call mps_mapping_free_memory after this
memory is already freed, causing a panic. Removed this extra
call to mps_mappiing_free_memory and call mps_mapping_exit
in place of the mps_mapping_free_memory call so that any
outstanding mapping items can be flushed before memory is
freed.

---------------------------------------------------------------
Correct memory leak during a Port reset with ouststanding I/O
---------------------------------------------------------------
In mps_reinit function, the mapping memory was not being
freed before being re-allocated. Added line to call the
memory free function for mapping memory.

---------------------------------------------------------------
Use CAM_SIM_QUEUED flag in Driver IO path.
---------------------------------------------------------------
This flag informs the XPT that successful abort of a CCB
requires an abort ccb to be issued to the SIM.  While
processing SCSI IO's, set the CAM_SIM_QUEUED flag in the
status for the IO. When the command completes, clear this
flag.

---------------------------------------------------------------
Check for CAM_REQ_INPROG in I/O path.
---------------------------------------------------------------
Added a check in mpssas_action_scsiio for the In Progress
status for the IO. If this flag is set, the IO has already
been aborted by the upper layer (before CAM_SIM_QUEUED was
set) and there is no need to send the IO. The request will
be completed without error.

---------------------------------------------------------------
Improve "doorbell handshake method" for mps_get_iocfacts
---------------------------------------------------------------
Removed call to get Port Facts since this information is
not used currently.

Added mps_iocfacts_allocate function to allocate memory
that is based on IOC Facts data.  Added mps_iocfacts_free
function to free memory that is based on IOC Facts data.
Both of the functions are used when a Diag Reset is performed
or when the driver is attached/detached. This is needed in
case IOC Facts changes after a Diag Reset, which could
happen if FW is upgraded.

Moved call of mps_bases_static_config_pages from the attach
routine to after the IOC is ready to process accesses based
on the new memory allocations (instead of polling through
the Doorbell).

---------------------------------------------------------------
Set TimeStamp in INIT message in millisecond format Set the IOC
---------------------------------------------------------------

---------------------------------------------------------------
Prefer mps_wait_command to mps_request_polled
---------------------------------------------------------------
Instead of using mps_request_polled, call mps_wait_command
whenever possible. Change the mps_wait_command function to
check the current context and either use interrupt context
or poll if required by using the pause or DELAY function.
Added a check after waiting 50mSecs to see if the command
has timed out. This is only done if polliing, the msleep
command will automatically timeout if the command has taken
too long to complete.

---------------------------------------------------------------
Integrated RAID: Volume Activation Failed error message is
displayed though the volume has been activated.
---------------------------------------------------------------
Instead of failing an IOCTL request that does not have a
large enough buffer to hold the complete reply, copy as
much data from the reply as possible into the user's buffer
and log a message saying that the user's buffer was smaller
than the returned data.

---------------------------------------------------------------
mapping_add_new_device failure due to persistent table FULL
---------------------------------------------------------------
When a new device is added, if it is determined that the
device persistent table is being used and is full, instead
of displaying a message for this condition every time, only
log a message if the MPS_INFO bit is set in the debug_flags.

Submitted by:	LSI
MFC after:	1 week
2013-07-22 18:41:53 +00:00
Scott Long
1610f95c56 Overhaul error, information, and debug logging.
Obtained from:	Netflix
MFC after:	3 days
2013-07-19 00:12:41 +00:00
Konstantin Belousov
dd0b4fb6d5 Reform the busdma API so that new types may be added without modifying
every architecture's busdma_machdep.c.  It is done by unifying the
bus_dmamap_load_buffer() routines so that they may be called from MI
code.  The MD busdma is then given a chance to do any final processing
in the complete() callback.

The cam changes unify the bus_dmamap_load* handling in cam drivers.

The arm and mips implementations are updated to track virtual
addresses for sync().  Previously this was done in a type specific
way.  Now it is done in a generic way by recording the list of
virtuals in the map.

Submitted by:	jeff (sponsored by EMC/Isilon)
Reviewed by:	kan (previous version), scottl,
	mjacob (isp(4), no objections for target mode changes)
Discussed with:	     ian (arm changes)
Tested by:	marius (sparc64), mips (jmallet), isci(4) on x86 (jharris),
	amd64 (Fabian Keil <freebsd-listen@fabiankeil.de>)
2013-02-12 16:57:20 +00:00
Alexander Motin
61c49b4dd1 Several fixes to allow firmware/BIOS flash access from user-level:
- remove special handling of zero length transfers in mpi_pre_fw_upload();
 - add missing MPS_CM_FLAGS_DATAIN flag in mpi_pre_fw_upload();
 - move mps_user_setup_request() call into proper place;
 - increase user command timeout from 30 to 60 seconds;
 - avoid NULL dereference panic in case of firmware crash.
Set max DMA segment size to 24bit, as MPI SGE supports it.
Use mps_add_dmaseg() to add empty SGE instead of custom code.
Tune endianness safety.

Reviewed by:	Desai, Kashyap <Kashyap.Desai@lsi.com>
Sponsored by:	iXsystems, Inc.
2012-08-01 17:31:31 +00:00
Kenneth D. Merry
be4aa869c1 Bring in LSI's latest mps(4) 6Gb SAS and WarpDrive driver, version
14.00.00.01-fbsd.

Their description of the changes is as follows:

1.	Copyright contents has been changed in all respective .c
	and .h files

2.	Support for WRITE12 and READ12 for direct-io (warpdrive only)
	has been added.

3.      Driver has added checks to see if Drive has READ_CAP_16
	support before sending it down to the device.
	If SPC3_SID_PROTECT flag is set in the inquiry data, the
	device supports protection information, and must support
	the 16 byte read capacity command, otherwise continue without
	sending read cap 16. This will optimize driver performance,
	since it will not send READ_CAP_16 to the drive which does
	not have support of READ_CAP_16.

4.      With new approach, "MPTIOCTL_RESET_ADAPTER" IOCTL will not
	use DELAY() which is busy loop implementation.
	It will use <msleep> (Better way to sleep without busy
	loop). Also from the HBA reset code path and some other
	places, DELAY() is replaced with msleep() or "pause()",
	which is based on sleep/wakeup style calls.  Driver use
	msleep()/pause() instead of DELAY based on CAN_SLEEP/NO_SLEEP
	flags to avoid busy loop which is not required all the
	time.e.a

	a. While driver is getting loaded, driver calls most of the
	   commands with NO_SLEEP.
	b. When Driver is functional and it needs Reinit of HBA,
	   CAN_SLEEP flag is used.

5.	<mpslsi> driver is not Endian safe. It will not work on Big
	Endian machines	like Sparc and PowerPC platforms because it
	assumes it is running on a Little Endian machine.

	Driver code is modified such way that it does not assume CPU
	arch is Little Endian.
	a. All places where Driver interacts from HBA to Host, it
	   converts Little Endian format to CPU format.
	b. All places where Driver interacts from Host to HBA, it
	   converts CPU format to Little Endian.

6.	Findout memory leaks in FreeBSD Driver and resolve those,
	such as memory leak in targ's luns creation/deletion.
	Also added additional checks to see memory allocation
	success/fail.

7.	Add loginfo prints as debug message, i.e. When FW sends any
	loginfo, Driver should print those as debug message.
	This will help for debugging purpose.

8.	There is possibility to get config request timeout. Current
	driver is able to detect config request timetout, but it does
	not do anything on config_request timeout.  Driver should
	call mps_reinit() if any request_poll (which is called as
	part of config_request) is time out.

9.	cdb length check is required for 32 byte CDB. Add correct mpi
	control value for 32 bit CDB as below while submitting SCSI IO
	Request to controller.
	mpi_control |= 4 << MPI2_SCSIIO_CONTROL_ADDCDBLEN_SHIFT;

10.	Check the actual status of Message unit reset
	(mps_message_unit_reset).Previously FreeBSD Driver just writes
	MPI2_FUNCTION_IOC_MESSAGE_UNIT_RESET and never check the ack
	(it just wait for 50 millisecond).  So, Driver now check the
	status of "MPI2_FUNCTION_IOC_MESSAGE_UNIT_RESET" after writing
	it to the FW.

	Now it also checking for whether doorbell ack uses msleep with
	proper sleep flags, instead of <DELAY>.

11.	Previously CAM does not detect Multi-Lun Devices. In order to
	detect Multi-Lun Devices by CAM the driver needs following change
	set:
	a. There is "max_lun" field which Driver need to set based on
	   hw/fw support. Currently LSI released driver does not set
	   this field.
	b. Default of "max_lun" should not be 0 in OS, but it is
	   currently set to 0 in CAM layer.
	c. Export max_lun capacity to 255

12.	Driver will not reset target info after port enable complete and
	also do Device removal when Device remove from FW.  The detail
	description is as follows
	a. When Driver receive WD PD add events, it will add all
	   information in driver local data structure.
	b. Only for WD, we have below checks after port enable
	   completes, where driver clear off all information retrieved
	   at #1.
	if ((sc->WD_available &&
             (sc->WD_hide_expose == MPS_WD_HIDE_ALWAYS)) ||
             (sc->WD_valid_config && (sc->WD_hide_expose ==
                            MPS_WD_HIDE_IF_VOLUME)) {
		  // clear off target data structure.
	}
	It is mainly not to attach PDs to OS.

	FreeBSD does bus rescan as older Parallel scsi style. So Driver
	needs to handle which Drive is visible to OS.  That is a reason
	we have to clear off targ information for PDs.

	Again, above logic was implemented long time ago. Similar concept
	we have for non-wd also. For that, LSI have introduced different
	logic to hide PDs.

	Eventually, because of above gap, when Phy goes offline, we
	observe below failure. That is what Driver is not doing complete
	removal of device with FW. (which was pointed by Scott)
	Apr  5 02:39:24 Freebsd7 kernel: mpslsi0: mpssas_prepare_remove
	Apr  5 02:39:24 Freebsd7 kernel: mpssas_prepare_remove 497 : invalid handle 0xe

	Now Driver will not reset target info after port enable complete
	and also will do Device removal when Device remove from FW.

13.	Returning "CAM_SEL_TIMEOUT" instead of "CAM_TID_INVALID"
	error code on request to the Target IDs that have no devices
	conected at that moment.  As if "CAM_TID_INVALID" error code
	is returned to the CAM Layaer then it results in a huge chain
	of errors in verbose kernel messages on boot and every
	hot-plug event.

Submitted by:	Sreekanth Reddy <Sreekanth.Reddy@lsi.com>
MFC after:	3 days
2012-06-28 03:48:54 +00:00
Kenneth D. Merry
653c521f8d Bring in a number of mps(4) driver fixes from LSI:
1.  Fixed timeout specification for the msleep in mps_wait_command().
    Added 30 second timeout for mps_wait_command() calls in mps_user.c.

2.  Make sure we call mps_detach_user() from the kldunload path.

3.  Raid Hotplug behavior change.

    The driver now removes a volume when it goes to a failed state,
    so we also need to add volume back to the OS when it goes to
    opitimal/degraded/online from failed/missing.

    Handle raid volume add and remove from the IR_Volume event.
4.  Added some more debugging information.

5.  Replace xpt_async(AC_LOST_DEVICE, path, NULL) with
    mpssas_rescan_target().

    This is to work around a panic in CAM that shows up when adding a
    drive with a rescan and removing another device from the driver thread
    with an AC_LOST_DEVICE async notification.

    This problem was encountered in testing with the LSI sas2ircu utility,
    which was used to create a RAID volume from physical disks.  The driver
    has to create the RAID volume target and remove the physical disk
    targets, and triggered a panic in the process.

    The CAM issue needs to be fully diagnosed and fixed, but this works
    around the issue for now.

6.  Fix some memory initialization issues in mps_free_command().

7.  Resolve the "devq freeze forever" issue.  This was caused by the
    internal read capacity command issued in the non-head version of the
    driver.  When the command completed with an error, the driver wasn't
    unfreezing thd device queue.

    The version in head uses the CAM infrastructure for getting the read
    capacity information, and therefore doesn't have the same issue.

8.  Bump the version to 13.00.00.00-fbsd. (this is very close to LSI's
    internal stable driver 13.00.00.00)

Submitted by:	Kashyap Desai <Kashyap.Desai@lsi.com>
MFC after:	3 days
2012-02-09 00:16:12 +00:00
Kenneth D. Merry
d043c56453 Bring in the LSI-supported version of the mps(4) driver.
This involves significant changes to the mps(4) driver, but is not a
complete rewrite.

Some of the changes in this version of the driver:
 - Integrated RAID (IR) support.
 - Support for WarpDrive controllers.
 - Support for SCSI protection information (EEDP).
 - Support for TLR (Transport Level Retries), needed for tape drives.
 - Improved error recovery code.
 - ioctl interface compatible with LSI utilities.

mps.4:		Update the mps(4) driver man page somewhat for the driver
		changes.  The list of supported hardware still needs to be
		updated to reflect the full list of supported cards.

conf/files:	Add the new driver files.

mps/mpi/*:	Updated version of the MPI header files, with a BSD style
		copyright.

mps/*:		See above for a description of the new driver features.

modules/mps/Makefile:
		Add the new mps(4) driver files.

Submitted by:	Kashyap Desai <Kashyap.Desai@lsi.com>
Reviewed by:	ken
MFC after:	1 week
2012-01-26 18:17:21 +00:00
Kenneth D. Merry
750ffe843d Silence 'out of chain frames' warnings and bump the number of frames.
mps.c:		Hide the 'out of chain frames' warning behind MPS_INFO.

mps_sas.c:	Hide the SIM queue freeze/unfreeze messages behind MPS_INFO.

mpsvar.h:	Bump the number of chain frames from 1024 to 2048.  From
		testing, it looks like this makes it less likely that we'll
		run out of chain frames, and it doesn't cost much memory
		(32K).

MFC after:	3 days
2011-02-25 17:30:25 +00:00
Kenneth D. Merry
550e2acdfc Fix several issues with the mps(4) driver.
When the driver ran out of DMA chaining buffers, it kept the timeout for
the I/O, and I/O would stall.

The driver was not freezing the device queue on errors.

mps.c:		Pull command completion logic into a separate
		function, and call the callback/wakeup for commands
		that are never sent due to lack of chain buffers.

		Add a number of extra diagnostic sysctl variables.

		Handle pre-hardware errors for configuration I/O.
		This doesn't panic the system, but it will fail the
		configuration I/O and there is no retry mechanism.
		So the device probe will not succeed.  This should
		be a very uncommon situation, however.

mps_sas.c:	Freeze the SIM queue when we run out of chain
		buffers, and unfreeze it when more commands
		complete.

		Freeze the device queue when errors occur, so that
		CAM can insure proper command ordering.

		Report pre-hardware errors for task management
		commands.  In general, that shouldn't be possible
		because task management commands don't have S/G
		lists, and that is currently the only error path
		before we get to the hardware.

		Handle pre-hardware errors (like out of chain
		elements) for SMP requests.  That shouldn't happen
		either, since we should have enough space for two
		S/G elements in the standard request.

		For commands that end with
		MPI2_IOCSTATUS_SCSI_IOC_TERMINATED and
		MPI2_IOCSTATUS_SCSI_EXT_TERMINATED, return them
		with CAM_REQUEUE_REQ to retry them unconditionally.
		These seem to be related to back end, transport
		related problems that are hopefully transient.  We
		don't want to go through the retry count for
		something that is not a permanent error.

		Keep track of the number of outstanding I/Os.

mpsvar.h:	Track the number of free chain elements.

		Add variables for the number of outstanding I/Os,
		and I/O high water mark.

		Add variables to track the number of free chain
		buffers and the chain low water mark, as well as
		the number of chain allocation failures.

		Add I/O state flags and an attach done flag.

MFC after:	3 days
2011-02-18 17:06:06 +00:00
Kenneth D. Merry
9866848a62 In the MPS driver, during device removal processing, don't assume that
the controller firmware will return all of our commands.  Instead, keep
track of outstanding I/Os and return them to CAM once device removal
processing completes.

mpsvar.h:	Declare the new "io_list" in the mps_softc.

mps.c:		Initialize the new "io_list" in the mps softc.

mps_sas.c:	o Track SCSI I/O requests on the io_list from the
		  time of mpssas_action() through mpssas_scsiio_complete().
		o Zero out the request structures used for device
		  removal commands prior to filling them out.
		o Once the target reset task management function completes
		  during device removal processing, assume any SCSI I/O
		  commands that are still oustanding will never return
		  from the controller, and process them manually.

Submitted by:	gibbs
MFC after:	3 days
2011-02-18 17:01:57 +00:00
Kenneth D. Merry
49dfe4a2d4 Fix an event handling bug with the mps(4) driver.
This bug manifested itself after repeated device arrivals and
departures.  The root of the problem was that the last entry in the
reply array wasn't initialized/allocated.  So every time we got
around to that event, we had a bogus address.

There were a couple more problems with the code that are also fixed:

 - The reply mechanism was being treated as sequential (indexed by
   sc->replycurindex) even though the spec says that the driver
   should use the ReplyFrameAddress field of the post queue
   descriptor to figure out where the reply is.  There is no
   guarantee that the reply descriptors will be used in sequential
   order.

 - The second word of the reply post queue descriptor wasn't being
   checked in mps_intr_locked() to make sure that it wasn't
   0xffffffff.  So the driver could potentially come across a
   partially DMAed descriptor.

 - The number of replies allocated was one less than the actual
   size of the queue.  Instead, it was the size of the number of
   replies that can be used at one time.  (Which is one less than
   the size of the queue.)

mps.c:		When initializing the entries in the reply free
		queue, make sure we initialize the full number that
		we tell the chip we have (sc->fqdepth), not the
		number that can be used at any one time (sc->num_replies).

		When allocating replies, make sure we allocate the
		number of replies that we've told the chip exist,
		not just the number that can be used simultaneously.

		Use the ReplyFrameAddress field of the post queue
		descriptor to figure out which reply is being
		referenced.  This is what the spec says to do, and
		the spec doesn't guarantee that the replies will be
		used in order.

		Put a check in to verify that the reply address passed
		back from the card is valid.  (Panic if it isn't, we'll
		panic when we try to deference the reply pointer in any
		case.)

		In mps_intr_locked(), verify that the second word of the
		post queue descriptor is not 0xffffffff in addition to
		verifying that the unused flag is not set, so we can
		make sure we didn't get a partially DMAed descriptor.

		Remove references to sc->replycurindex, it isn't needed
		now.

mpsvar.h:	Remove replycurindex from the softc, it isn't needed now.

Reviewed by:	scottl
2010-12-10 21:45:10 +00:00
Kenneth D. Merry
06e794928b Add Serial Management Protocol (SMP) passthrough support to CAM.
This includes support in the kernel, camcontrol(8), libcam and the mps(4)
driver for SMP passthrough.

The CAM SCSI probe code has been modified to fetch Inquiry VPD page 0x00
to determine supported pages, and will now fetch page 0x83 in addition to
page 0x80 if supported.

Add two new CAM CCBs, XPT_SMP_IO, and XPT_GDEV_ADVINFO.  The SMP CCB is
intended for SMP requests and responses.  The ADVINFO is currently used to
fetch cached VPD page 0x83 data from the transport layer, but is intended
to be extensible to fetch other types of device-specific data.

SMP-only devices are not currently represented in the CAM topology, and so
the current semantics are that the SIM will route SMP CCBs to either the
addressed device, if it contains an SMP target, or its parent, if it
contains an SMP target.  (This is noted in cam_ccb.h, since it will change
later once we have the ability to have SMP-only devices in CAM's topology.)

smp_all.c,
smp_all.h:		New helper routines for SMP.  This includes
			SMP request building routines, response parsing
			routines, error decoding routines, and structure
			definitions for a number of SMP commands.

libcam/Makefile:	Add smp_all.c to libcam, so that SMP functionality
			is available to userland applications.

camcontrol.8,
camcontrol.c:		Add smp passthrough support to camcontrol.  Several
			new subcommands are now available:

			'smpcmd' functions much like 'cmd', except that it
			allows the user to send generic SMP commands.

			'smprg' sends the SMP report general command, and
			displays the decoded output.  It will automatically
			fetch extended output if it is available.

			'smppc' sends the SMP phy control command, with any
			number of potential options.  Among other things,
			this allows the user to reset a phy on a SAS
			expander, or disable a phy on an expander.

			'smpmaninfo' sends the SMP report manufacturer
			information and displays the decoded output.

			'smpphylist' displays a list of phys on an
			expander, and the CAM devices attached to those
			phys, if any.

cam.h,
cam.c:			Add a status value for SMP errors
			(CAM_SMP_STATUS_ERROR).

			Add a missing description for CAM_SCSI_IT_NEXUS_LOST.

			Add support for SMP commands to cam_error_string().

cam_ccb.h:		Rename the CAM_DIR_RESV flag to CAM_DIR_BOTH.  SMP
			commands are by nature bi-directional, and we may
			need to support bi-directional SCSI commands later.

			Add the XPT_SMP_IO CCB.  Since SMP commands are
			bi-directional, there are pointers for both the
			request and response.

			Add a fill routine for SMP CCBs.

			Add the XPT_GDEV_ADVINFO CCB.  This is currently
			used to fetch cached page 0x83 data from the
			transport later, but is extensible to fetch many
			other types of data.

cam_periph.c:		Add support in cam_periph_mapmem() for XPT_SMP_IO
			and XPT_GDEV_ADVINFO CCBs.

cam_xpt.c:		Add support for executing XPT_SMP_IO CCBs.

cam_xpt_internal.h:	Add fields for VPD pages 0x00 and 0x83 in struct
			cam_ed.

scsi_all.c:		Add scsi_get_sas_addr(), a function that parses
			VPD page 0x83 data and pulls out a SAS address.

scsi_all.h:		Add VPD page 0x00 and 0x83 structures, and a
			prototype for scsi_get_sas_addr().

scsi_pass.c:		Add support for mapping buffers in XPT_SMP_IO and
			XPT_GDEV_ADVINFO CCBs.

scsi_xpt.c:		In the SCSI probe code, first ask the device for
			VPD page 0x00.  If any VPD pages are supported,
			that page is required to be implemented.  Based on
			the response, we may probe for the serial number
			(page 0x80) or device id (page 0x83).

			Add support for the XPT_GDEV_ADVINFO CCB.

sys/conf/files:		Add smp_all.c.

mps.c:			Add support for passing in a uio in mps_map_command(),
			so we can map a S/G list at once.

			Add support for SMP passthrough commands in
			mps_data_cb().  SMP is a special case, because the
			first buffer in the S/G list is outbound and the
			second buffer is inbound.

			Add support for warning the user if the busdma code
			comes back with more buffers than will work for the
			command.  This will, for example, help the user
			determine why an SMP command failed if busdma comes
			back with three buffers.

mps_pci.c:		Add sys/uio.h.

mps_sas.c:		Add the SAS address and the parent handle to the
			list of fields we pull from device page 0 and cache
			in struct mpssas_target.  These are needed for SMP
			passthrough.

			Add support for the XPT_SMP_IO CCB.  For now, this
			CCB is routed to the addressed device if it supports
			SMP, or to its parent if it does not and the parent
			does.  This is necessary because CAM does not
			currently support SMP-only nodes in the topology.

			Make SMP passthrough support conditional on
			__FreeBSD_version >= 900026.  This will make it
			easier to MFC this change to the driver without
			MFCing the CAM changes as well.

mps_user.c:		Un-staticize mpi_init_sge() so we can use it for
			the SMP passthrough code.

mpsvar.h:		Add a uio and iovecs into struct mps_command for
			SMP passthrough commands.

			Add a cm_max_segs field to struct mps_command so
			that we can warn the user if busdma comes back with
			too many segments.

			Clear the cm_reply when a command gets freed.  If
			it is not cleared, reply frames will eventually get
			freed into the pool multiple times and corrupt the
			pool.  (This fix is from scottl.)

			Add a prototype for mpi_init_sge().

sys/param.h:		Bump __FreeBSD_version to 900026 for the for the
			inclusion of the XPT_GDEV_ADVINFO and XPT_SMP_IO
			CAM CCBs.
2010-11-30 22:39:46 +00:00
Matthew D Fleming
13ac67a784 Re-work the internals of adding items to the driver's scatter-gather
list.  Use the new internals to simplify adding transaction context
elements, and in future diffs, more complicated SGLs.
2010-10-14 16:44:05 +00:00
Kenneth D. Merry
1476ba4087 Turn on serialization of task management commands going down to the
controller, but make it optional.

After a problem report from Andrew Boyer, it looks like the LSI
chip may have issues (the watchdog timer fired) if too many aborts
are sent down to the chip at the same time.  We know that task
management commands are serialized, and although the manual doesn't
say it, it may be a good idea to just send one at a time.

But, since I'm not certain that this is necessary, add a tunable
and sysctl variable (hw.mps.%d.allow_multiple_tm_cmds) to control
the driver's behavior.

mps.c:		Add support for the sysctl and tunable, and add a
		comment about the possible return values to
		mps_map_command().

mps_sas.c:	Run all task management commands through two new
		routines, mpssas_issue_tm_request() and
		mpssas_complete_tm_request().

		This allows us to optionally serialize task
		management commands.  Also, change things so that
		the response to a task management command always
		comes back through the callback.  (Before it could
		come via the callback or the return value.)

mpsvar.h:	Add softc variables for the list of active task
		management commands, the number of active commands,
		and whether we should allow multiple active task
		management commands.  Add an active command flag.

mps.4:		Describe the new sysctl/loader tunable variable.

Sponsored by:	Spectra Logic Corporation
2010-10-07 21:56:10 +00:00
Kenneth D. Merry
d3c7b9a08a MFp4 (//depot/projects/mps/...)
Bring in a driver for the LSI Logic MPT2 6Gb SAS controllers.

This driver supports basic I/O, and works with SAS and SATA drives and
expanders.

Basic error recovery works (i.e. timeouts and aborts) as well.

Integrated RAID isn't supported yet, and there are some known bugs.

So this isn't ready for production use, but is certainly ready for
testing and additional development.  For the moment, new commits to this
driver should go into the FreeBSD Perforce repository first
(//depot/projects/mps/...) and then get merged into -current once
they've been vetted.

This has only been added to the amd64 GENERIC, since that is the only
architecture I have tested this driver with.

Submitted by:	scottl
Discussed with:	imp, gibbs, will
Sponsored by:	Yahoo, Spectra Logic Corporation
2010-09-10 15:03:56 +00:00