All resources lack of which may put CTIO into the queue are either
per-channel or potentially per-queue, but none of them are per-LUN.
This is a first step to fix live LUN disabling. Before this change
any CTIOs held in a queue in time of disabling were just leaked.
MFC after: 2 weeks
Usually IOCBs should be put on queue for asynchronous processing and should
not require additional DMA memory. But there are some cases like aborts and
resets that for external reasons has to be synchronous. Give those cases
separate 2*64 byte DMA area to decouple them from other DMA scratch area
users, using it for asynchronous requests.
This is cosmetics that simplifies identification of new ports on FC switch.
It would be good to use target name from CTL here instead of hostname, but
it is not passed here through CAM now.
MFC after: 2 weeks
- Make scan aborted by event restart immediately and infinitely.
- Improve handling of some loop events from firmware.
- Remove loop down timer, adding its functionality to scanner thread.
- Some more unification and simplification.
Hacks to enable target mode there complicated code, while didn't really
work. And for outdated hardware fixing it is not really interesting.
Initiator mode tested with Qlogic 1080 adapter is still working fine.
Now 24xx and above chips support full 8-byte LUN address space.
Older FC chips may support up to 16K LUNs when firmware allows.
Tested in both initiator and target modes for 23xx, 24xx and 25xx.
FC port database code already notifies CAM about all devices. Additional
full scan is just a waste of time, that by definition won't find anything
that is not present in port database.
The Command Reference Number is used for precise delivery of
commands, and is part of the FC-Tape functionality set. (This is
only enabled for devices that support precise delivery of commands.)
It is an 8-bit unsigned number that increments from 1 to 255. The
commands sent by the initiator must be processed by the target in
CRN order if the CRN is non-zero.
There are certain scenarios where the Command Reference Number
sequence needs to be reset. When the target is power cycled, for
instance, the initiator needs to reset the CRN to 1. The initiator
will know this because it will see a LIP (when directly connected)
or get a logout/login event (when connected to a switch).
The isp(4) driver was not resetting the CRN when a target
went away and came back. When it saw the target again after a
power cycle, it would continue the CRN sequence where it left off.
The target would ignore the command because the CRN sequence is
supposed to be reset to 1 after a power cycle or other similar
event.
The symptom that the user would see is that there would be lots of
aborted INQUIRY commands after a tape library was power cycled, and
the library would fail to probe. The INQUIRY commands were being
ignored by the tape drive due to the CRN issue mentioned above.
isp_freebsd.c:
Add a new function, isp_fcp_reset_crn(). This will reset
all of the CRNs for a given port, or the CRNs for all LUNs
on a target.
Reset the CRNs for all targets on a port when we get a LIP,
loop reset, or loop down event.
Reset the CRN for a particular target when it arrives, is changed
or departs. This is less precise behavior than the
clearing behavior specified in the FCP-4 spec (which says
that it should be reset for PRLI, PRLO, PLOGI and LOGO),
but this is the level of information we have here. If this
is insufficient, then we will need to add more precise
notification from the lower level isp(4) code.
isp_freebsd.h:
Add a prototype for isp_fcp_reset_crn().
Sponsored by: Spectra Logic
MFC after: 1 week
In the current implementation, the isp_kthread() threads never exit.
The target threads do have an exit mode from isp_attach(), but it is
not invoked from isp_detach().
Ensure isp_detach() notifies threads started for each channel, such
that they exit before their parent device softc detaches, and thus
before the module does. Otherwise, a page fault panic occurs later in:
sysctl_kern_proc
sysctl_out_proc
kern_proc_out
fill_kinfo_proc
fill_kinfo_thread
strlcpy(kp->ki_wmesg, td->td_wmesg, sizeof(kp->ki_wmesg));
For isp_kthread() (and isp(4) target threads), td->td_wmesg references
now-unmapped memory after the module has been unloaded. These threads
are typically msleep()ing at the time of unload, but they could also
attempt to execute now-unmapped code segments.
MFC after: 1 month
Sponsored by: Spectra Logic
MFSpectraBSD: r1070921 on 2014/06/22 13:01:17
mostly by adjustments to debugging printf() format specifiers. For high
numbered LUNs, also switch to printing them in hex as per SAM-5.
MFC after: 2 weeks
- Remove two excessive and slow register reads from isp_intr(). Instead
of rereading value every time, assume that registers contain what we have
written there.
- Avoid sequential search through 4096 array elements when looking for
command tag. Use hash of lists to store active tags separately from free
ones and so greatly speedup the searches.
Reviewed by: mjacob
CCB at a time outstanding reliable. It's not there yet, but this
is the direction to go in so might as well commit. So far,
multiple at a time CCBs work (see ISP_INTERNAL_TARGET test mode),
but it fails if there are more downstream than the SIM wants
to handle and SRR is sort of confused when this happens, plus
it is not entirely quite clear what one does if a CCB/CTIO fails
and you have more in flight (that don't fail, say) and more queued
up at the SIM level that haven't been started yet.
Some of this is driven because there apparently is no flow control
to requeue XPT_CONTINUE_IO requests like there are for XPT_SCSI_IO
requests. It is also more driven in that the few target mode
periph drivers there are are not really set up for handling pushback-
heck most of them don't even check for errors (and what would they
really do with them anyway? It's the initiator's problem, really....).
The data transfer arithmetic has been worked over again to handle
multiple outstanding commands, so you have a notion of what's been
moved already as well as what's currently in flight. It turns that
this led to uncovering a REPORT_LUNS bug in the ISP_INTERNAL_TARGET
code which was sending back 24 bytes of rpl data instead of the
specified 16. What happened furthermore here is that sending back
16 bytes and reporting an overrun of 8 bytes made the initiator
(running FC-Tape aware f/w) mad enough to request, and keep
requesting, another FCP response (I guess it didn't like the answer
so kept asking for it again).
Sponsored by: Spectralogic
MFC after: 1 month
MISC CHANGES
Add a new async event- ISP_TARGET_NOTIFY_ACK, that will guarantee
eventual delivery of a NOTIFY ACK. This is tons better than just
ignoring the return from isp_notify_ack and hoping for the best.
Clean up the lower level lun enable code to be a bit more sensible.
Fix a botch in isp_endcmd which was messing up the sense data.
Fix notify ack for SRR to use a sensible error code in the case
of a reject.
Clean up and make clear what kind of firmware we've loaded and
what capabilities it has.
-----------
FULL (252 byte) SENSE DATA
In CTIOs for the ISP, there's only a limimted amount of space
to load SENSE DATA for associated CHECK CONDITIONS (24 or 26
bytes). This makes it difficult to send full SENSE DATA that can
be up to 252 bytes.
Implement MODE 2 responses which have us build the FCP Response
in system memory which the ISP will put onto the wire directly.
On the initiator side, the same problem occurs in that a command
status response only has a limited amount of space for SENSE DATA.
This data is supplemented by status continuation responses that
the ISP pushes onto the response queue after the status response.
We now pull them all together so that full sense data can be
returned to the periph driver.
This is supported on 23XX, 24XX and 25XX cards.
This is also preparation for doing >16 byte CDBs.
-----------
FC TAPE
Implement full FC-TAPE on both initiator and target mode side. This
capability is driven by firmware loaded, board type, board NVRAM
settings, or hint configuration options to enable or disable. This
is supported for 23XX, 24XX and 25XX cards.
On the initiator side, we pretty much just have to generate a command
reference number for each command we send out. This is FCP-4 compliant
in that we do this per ITL nexus to generate the allowed 1 thru 255
CRN.
In order to support the target side of FC-TAPE, we now pay attention
to more of the PRLI word 3 parameters which will tell us whether
an initiator wants confirmed responses. While we're at it, we'll
pay attention to the initiator view too and report it.
On sending back CTIOs, we will notice whether the initiator wants
confirmed responses and we'll set up flags to do so.
If a response or data frame is lost the initiator sends us an SRR
(Sequence Retransmit Request) ELS which shows up as an SRR notify
and all outstanding CTIOs are nuked with SRR Received status. The
SRR notify contains the offset that the initiator wants us to restart
the data transfer from or to retransmit the response frame.
If the ISP driver still has the CCB around for which the data segment
or response applies, it will retransmit.
However, we typically don't know about a lost data frame until we
send the FCP Response and the initiator totes up counters for data
moved and notices missing segments. In this case we've already
completed the data CCBs already and sent themn back up to the periph
driver. Because there's no really clean mechanism yet in CAM to
handle this, a hack has been put into place to complete the CTIO
CCB with the CAM_MESSAGE_RECV status which will have a MODIFY DATA
POINTER extended message in it. The internal ISP target groks this
and ctl(8) will be modified to deal with this as well.
At any rate, the data is retransmitted and an an FCP response is
sent. The whole point here is to successfully complete a command
so that you don't have to depend on ULP (SCSI) to have to recover,
which in the case of tape is not really possible (hence the name
FC-TAPE).
Sponsored by: Spectralogic
MFC after: 1 month
not by some hint setting. Do more preparations for FC-Tape.
Clean up resource counting for 24XX or later chipsets so
we find out after EXEC_FIRMWARE what is actually supported.
Set target mode exchange count based upon whether or not
we are supporting simultaneous target/initiator mode. Clean
up some old (pre-24XX) xfwoption and zfwoption issues.
Sponsored by: Spectralogic
MFC after: 3 days
and crosschecks against firmware documentation. We now check and report
FC firmware attributes and at least are now prepared for the upper 48 bits
of f/w attributes (which are probably for the 8100 or later cards). This
involed changing how inbits and outbits are calculated for varios commands,
hopefully clearer and cleaner. This also caused me to clean up the actual
mailbox register usage. Finally, we are now unconditionally using a CRN
for initiator mode.
A longstanding issue with the 2400/2500 is that they do *not* support
a "Prefer PTP followed by loop", which explains why enabling that
caused the f/w to crash.
A slightly more invasive change is to let the firmware load entirely
drive whether multi_id support is enabled or not.
Sponsored by: Spectralogic
MFC after: 1 week
Make the default role NONE if target mode is selected. This
allows ctl(8) to switch to/from target mode via knob settings.
If we default to role 'none', this causes a reset of the
24XX f/w which then causes initiators to wake up and notice
when we come online.
Reviewed by: kdm
MFC after: 2 weeks
Sponsored by: Spectralogic
Zero any sense not transferred by the device as the SCSI specification
mandates that any untransferred data should be assumed to be zero.
Reviewed by: ken
CAM.
Desriptor sense is a new sense data format that originated in SPC-3. Among
other things, it allows for an 8-byte info field, which is necessary to
pass back block numbers larger than 4 bytes.
This change adds a number of new functions to scsi_all.c (and therefore
libcam) that abstract out most access to sense data.
This includes a bump of CAM_VERSION, because the CCB ABI has changed.
Userland programs that use the CAM pass(4) driver will need to be
recompiled.
camcontrol.c: Change uses of scsi_extract_sense() to use
scsi_extract_sense_len().
Use scsi_get_sks() instead of accessing sense key specific
data directly.
scsi_modes: Update the control mode page to the latest version (SPC-4).
scsi_cmds.c,
scsi_target.c: Change references to struct scsi_sense_data to struct
scsi_sense_data_fixed. This should be changed to allow the
user to specify fixed or descriptor sense, and then use
scsi_set_sense_data() to build the sense data.
ps3cdrom.c: Use scsi_set_sense_data() instead of setting sense data
manually.
cam_periph.c: Use scsi_extract_sense_len() instead of using
scsi_extract_sense() or accessing sense data directly.
cam_ccb.h: Bump the CAM_VERSION from 0x15 to 0x16. The change of
struct scsi_sense_data from 32 to 252 bytes changes the
size of struct ccb_scsiio, but not the size of union ccb.
So the version must be bumped to prevent structure
mis-matches.
scsi_all.h: Lots of updated SCSI sense data and other structures.
Add function prototypes for the new sense data functions.
Take out the inline implementation of scsi_extract_sense().
It is now too large to put in a header file.
Add macros to calculate whether fields are present and
filled in fixed and descriptor sense data
scsi_all.c: In scsi_op_desc(), allow the user to pass in NULL inquiry
data, and we'll assume a direct access device in that case.
Changed the SCSI RESERVED sense key name and description
to COMPLETED, as it is now defined in the spec.
Change the error recovery action for a number of read errors
to prevent lots of retries when the drive has said that the
block isn't accessible. This speeds up reconstruction of
the block by any RAID software running on top of the drive
(e.g. ZFS).
In scsi_sense_desc(), allow for invalid sense key numbers.
This allows calling this routine without checking the input
values first.
Change scsi_error_action() to use scsi_extract_sense_len(),
and handle things when invalid asc/ascq values are
encountered.
Add a new routine, scsi_desc_iterate(), that will call the
supplied function for every descriptor in descriptor format
sense data.
Add scsi_set_sense_data(), and scsi_set_sense_data_va(),
which build descriptor and fixed format sense data. They
currently default to fixed format sense data.
Add a number of scsi_get_*() functions, which get different
types of sense data fields from either fixed or descriptor
format sense data, if the data is present.
Add a number of scsi_*_sbuf() functions, which print
formatted versions of various sense data fields. These
functions work for either fixed or descriptor sense.
Add a number of scsi_sense_*_sbuf() functions, which have a
standard calling interface and print the indicated field.
These functions take descriptors only.
Add scsi_sense_desc_sbuf(), which will print a formatted
version of the given sense descriptor.
Pull out a majority of the scsi_sense_sbuf() function and
put it into scsi_sense_only_sbuf(). This allows callers
that don't use struct ccb_scsiio to easily utilize the
printing routines. Revamp that function to handle
descriptor sense and use the new sense fetching and
printing routines.
Move scsi_extract_sense() into scsi_all.c, and implement it
in terms of the new function, scsi_extract_sense_len().
The _len() version takes a length (which should be the
sense length - residual) and can indicate which fields are
present and valid in the sense data.
Add a couple of new scsi_get_*() routines to get the sense
key, asc, and ascq only.
mly.c: Rename struct scsi_sense_data to struct
scsi_sense_data_fixed.
sbp_targ.c: Use the new sense fetching routines to get sense data
instead of accessing it directly.
sbp.c: Change the firewire/SCSI sense data transformation code to
use struct scsi_sense_data_fixed instead of struct
scsi_sense_data. This should be changed later to use
scsi_set_sense_data().
ciss.c: Calculate the sense residual properly. Use
scsi_get_sense_key() to fetch the sense key.
mps_sas.c,
mpt_cam.c: Set the sense residual properly.
iir.c: Use scsi_set_sense_data() instead of building sense data by
hand.
iscsi_subr.c: Use scsi_extract_sense_len() instead of grabbing sense data
directly.
umass.c: Use scsi_set_sense_data() to build sense data.
Grab the sense key using scsi_get_sense_key().
Calculate the sense residual properly.
isp_freebsd.h: Use scsi_get_*() routines to grab asc, ascq, and sense key
values.
Calculate and set the sense residual.
MFC after: 3 days
Sponsored by: Spectra Logic Corporation
We also revive loop down freezes. We also externaliz within isp
isp_prt_endcmd so something outside the core module can print
something about a command completing. Also some work in progress to
assist in handling timed out commands better.
Partially Sponsored by: Panasas
Approved by: re (kib)
MFC after: 1 month
once. Use taskqueues to do the actual work.
Fix an offset line.
Fix isp_prt so that prints from just one buffer, which makes it
appear cleanly cleanly in logs on SMP systems.
Approved by: re (kib)
MFC after: 1 month
- Allocate coherent DMA memory for the request/response queue area and
and the FC scratch area.
These changes allow isp(4) to work properly on sparc64 with usage of the
IOMMU streaming buffers enabled.
Approved by: mjacob
MFC after: 2 weeks
on debug output. Add a new platform function requirement to allow
for printing based upon the ITL nexus instead of the isp unit plus
channel, target and lun. This allows some printouts and error messages
from the core code to appear in the same format as the platform's
subsystem (in FreeBSD's case, CAM path).
MFC after: 1 week
Untangle some of the confusion about what role means when it's in the FCPARAM/SDPARAM
or isp_fc/isp_spi structures. This fixed a problem about seeing targets appear if you've
turned off autologin and find them, or rather don't, via camcontrol rescan.
MFC after: 1 month
the loop down counter, as well as other things. This was brought to my
attention with a different fix, more for RELENG_7- this one covers the
multiple channel case.
PR: 140438
MFC after: 1 month
firmware loading bugs.
Target mode support has received some serious attention to make it
more usable and stable.
Some backward compatible additions to CAM have been made that make
target mode async events easier to deal with have also been put
into place.
Further refinement and better support for NP-IV (N-port Virtualization)
is now in place.
Code for release prior to RELENG_7 has been stripped away for code clarity.
Sponsored by: Copan Systems
Reviewed by: scottl, ken, jung-uk kim
Approved by: re
modularize it so that new transports can be created.
Add a transport for SATA
Add a periph+protocol layer for ATA
Add a driver for AHCI-compliant hardware.
Add a maxio field to CAM so that drivers can advertise their max
I/O capability. Modify various drivers so that they are insulated
from the value of MAXPHYS.
The new ATA/SATA code supports AHCI-compliant hardware, and will override
the classic ATA driver if it is loaded as a module at boot time or compiled
into the kernel. The stack now support NCQ (tagged queueing) for increased
performance on modern SATA drives. It also supports port multipliers.
ATA drives are accessed via 'ada' device nodes. ATAPI drives are
accessed via 'cd' device nodes. They can all be enumerated and manipulated
via camcontrol, just like SCSI drives. SCSI commands are not translated to
their ATA equivalents; ATA native commands are used throughout the entire
stack, including camcontrol. See the camcontrol manpage for further
details. Testing this code may require that you update your fstab, and
possibly modify your BIOS to enable AHCI functionality, if available.
This code is very experimental at the moment. The userland ABI/API has
changed, so applications will need to be recompiled. It may change
further in the near future. The 'ada' device name may also change as
more infrastructure is completed in this project. The goal is to
eventually put all CAM busses and devices until newbus, allowing for
interesting topology and management options.
Few functional changes will be seen with existing SCSI/SAS/FC drivers,
though the userland ABI has still changed. In the future, transports
specific modules for SAS and FC may appear in order to better support
the topologies and capabilities of these technologies.
The modularization of CAM and the addition of the ATA/SATA modules is
meant to break CAM out of the mold of being specific to SCSI, letting it
grow to be a framework for arbitrary transports and protocols. It also
allows drivers to be written to support discrete hardware without
jeopardizing the stability of non-related hardware. While only an AHCI
driver is provided now, a Silicon Image driver is also in the works.
Drivers for ICH1-4, ICH5-6, PIIX, classic IDE, and any other hardware
is possible and encouraged. Help with new transports is also encouraged.
Submitted by: scottl, mav
Approved by: re
Both WWNN and WWPN are 64-bit unsigned integers and they are prefixed
with "0x", which requires two more bytes each.
Submitted by: Danny Braniss (danny at cs dot huji dot ac dot il)
via Matthew Jacob (lydianconcepts at gmail dot com)
Approved by: re (bmah)
MFC after: 3 days
First, we were never correctly checking for a 24XX Status Type 0
response- that cased us to fall through to evaluate status for
commands as if this were a 2100/2200/2300 Status Type 0 response.
This is *close*, but not quite the same. This has been reported
to be apparent with some wierd lun configuration problems with
some arrays. It became glaringly apparent on sparc64 where none
of the correct byte swap things were done.
Fixing this omission then caused a whole universe shifting debug
cycle of endian issues for the 2400. The manual for 24XX f/w turns
out to be wrong about the endianness of a couple of entities. The
lun and cdb fields for the type 7 request are *not* unconditionally
big endian- they happen to be opposite of whatever the endian of
the current machine type is. Same with the sense data for the
24XX type 0 response.
While we're at it investigate and resolve some NVRAM endian
issues.
Approved by: re (ken)
MFC after: 3 days
Seems to work on RELENG_4 through -current and also on sparc64
now. There may still be some issues with the auto attach/detach
code to sort out.
MFC after: 3 days
When the linux port changes were imported which split the
target command list to be separate from the initiator command
list and the handle format changed to encode a type in the handle
the implications to the function isp_handle_index (which only
the NetBSD/OpenBSD/FreeBSD ports use) were overlooked.
The fault is twofold: first, the index into the DMA maps
in isp_pci is wrong because a target command handle with
the type bit left in place caused a bad index (and panic)
into dma map. Secondly, the assumption of the array
of DMA maps in either PCS or SBUS attachment structures is
that there is a linear mapping between handle index and
DMA map index. This can no longer be true if there are
overlapping index spaces for initiator mode and target
mode commands.
These changes bandaid around the problem by forcing us
to not have simultaneous dual roles and doing the appropriate
masking to make sure things are indexed correctly. A longer
term fix is being devloped.
