sure that it really is by issuing a ISPCTL_ABORT_CMD just on the
off chance the f/w will start it up again and, ha ha, start using
the DMA resources we gave it but are now taking away.
us to not the ints are ok and also to (re)ENABLE isp interrupts. Remove
all splcam()/splx() invocates and replace them with ISP_LOCK/ISP_UNLOCK
macros.
to isp_osinfo substructure (all in prep for SMP). Define MBOX_WAIT_COMPLETE
and MBOX_NOTIFY_COMPLETE macros so that we can now (temp) use tsleep
to wait for mailbox completion. Requires us to guess whether we're
servicing an interrupt or not- will use intr_nesting_level.
Add local strncat function.
define). Fix stupidity wrt checking whether we've gone to
LOOP_PDB_RCVD loopstate- it's okay to be greater than this state.
D'oh! Protect calls to isp_pdb_sync and isp_fclink_state with IS_FC
macros.
Completely redo mailbox command routine (in preparation to make this
possibly wait rather than poll for completion).
Make a major attempt to solve the 'lost interrupt' problem
1. Problem
The Qlogic cards would appear to 'lose' interrupts, i.e., a legitimate
regular SCSI command placed on the request queue would never complete
and the watchdog routine in the driver would eventually wakeup and
catch it. This would typically only happen on Alphas, although a
couple folks with 700MHz Intel platforms have also seen this.
For a long time I thought it was a foulup with f/w negotiations of
SYNC and/or WIDE as it always seemed to happen right after the
platform it was running on had done a SET TARGET PARAMETERS mailbox
command to (re)enable sync && wide (after initially forcing
ASYNC/NARROW at startup). However, occasionally, the same thing
would also occur for the Fibre Channel cards as well (which, ahem,
have no SET TARGET PARAMETERS for transfer mode).
After finally putting in a better set of watchdog routines for the
platforms for this driver, it seemed to be the case that the command
in question (usually a READ CAPACITY) just had up and died- the
watchdog routine would catch it after ~10 seconds. For some platforms
(NetBSD/OpenBSD)- an ABORT COMMAND mailbox command was sent (which
would always fail- indicating that the f/w denied knowledge of this
command, i.e., the f/w thought it was a done command). In any case,
retrying the command worked. But this whole problem needed to be
really fixed.
2. A False Step That Went in The Right Direction
The mailbox code was completely rewritten to no longer try and grab
the mailbox semaphore register and to try and 'by hand' complete
async fast posting completions. It was also rewritten to now have
separate in && out bitpatterns for registers to load to start and
retrieve to complete. This means that isp_intr now handles mailbox
completions.
This substantially simplifies the mailbox handling code, and carries
things 90% toward getting this to be a non-polled routine for this
driver.
This did not solve the problem, though.
3. Register Debouncing
I saw some comments in some errata sheets and some notes in a Qlogic
produced Linux driver (for the Qlogic 2100) that seemed to indicate
that debouncing of reads of the mailbox registers might be needed,
so I added this. This did not affect the problem. In fact, it made
the problem worse for non-2100 cards.
5. Interrupt masking/unmasking
The driver *used* to do a substantial amount of masking/unmasking
of the interrupt control register. This was done to make sure that
the core common code could just assume it would never get pre-empted.
This apparently substantially contributed to the lost interrupt
problem. The rewrite of the ICR (Interrupt Control Register),
which is a separate register from the ISR (Interrupt Status Register)
should not have caused any change to interrupt assertions pending.
The manual does not state that it will, and the register layout
seems to imply that the ICR is just an active route gate. We only
enable PCI Interrupts and RISC Interrupts- this should mean that
when the f/w asserts a RISC interrupt and (and the ICR allows RISC
Interrupts) and we have PCI Interrupts enabled, we should get a
PCI interrupt. Apparently this is a latch- not a signal route.
Removing this got rid of *most* but not all, lost interrupts.
5. Watchdog Smartening
I made sure that the watchdog routine would catch cases where the
Qlogic's ISR showed an interrupt assertion. The watchdog routine
now calls the interrupt service routine if it sees this. Some
additional internal state flags were added so that the watchdog
routine could then know whether the command it was in the middle
of burying (because we had time it out) was in fact completed by
the interrupt service routine.
6. Occasional Constipation Of Commands..
In running some very strenous high IOPs tests (generating about
11000 interrupts/second across one Qlogic 1040, one Qlogic 1080
and one Qlogic 2200 on an Alpha PC164), I found that I would get
occasional but regular 'watchdog timeouts' on both the 1080 and
the 2100 cards. This is under FreeBSD, and the watchdog timeout
routine just marks the command in error and retries it.
Invariably, right after this 'watchdog timeout' error, I'd get a
command completion for the command that I had thought timed out.
That is, I'd get a command completion, but the handle returned by
the firmware mapped to no current command. The frequency of this
problem is low under such a load- it would usually take an 30
minutes per 'lost' interrupt.
I doubled the timeout for commands to see if it just was an edge
case of waiting too short a period. This has no effect.
I gathered and printed out microtimes for the watchdog completed
command and the completion that couldn't find a command- it was
always the case that the order of occurrence was "timeout, completion"
separated by a time on the order of 100 to 150 ms.
This caused me to consider 'firmware constipation' as to be a
possible culprit. That is, resubmission of a command to the device
that had suffered a watchdog timeout seemed to cause the presumed
dead command to show back up.
I added code in the watchdog routine that, when first entered for
the command, marks the command with a flag, reissues a local timeout
call for one second later, but also then issues a MARKER Request
Queue entry to the Qlogic f/w. A MARKER entry is used typically
after a Bus Reset to cause the f/w to get synchronized with respect
to either a Bus, a Nexus or a Target.
Since I've added this code, I always now see the occasional watchdog
timeout, but the command that was about to be terminated always
now seems to be completed after the MARKER entry is issued (and
before the timeout extension fires, which would come back and
*really* terminate the command).
comment. Check against firmware state- not loop state when enabling
target mode. Other changes have to do with no longer enabling/disabling
interrupts at will.
Rearchitect command watchdog timeouts-
First of all, set the timeout period for a command that has a
timeout (in isp_action) to the period of time requested *plus* two
seconds. We don't want the Qlogic firmware and the host system to
race each other to report a dead command (the watchdog is there to
catch dead and/or broken firmware).
Next, make sure that the command being watched isn't done yet. If
it's not done yet, check for INT_PENDING and call isp_intr- if that
said it serviced an interrupt, check to see whether the command is
now done (this is what the "IN WATCHDOG" private flag is for- if
isp_intr completes the command, it won't call xpt_done on it because
isp_watchdog is still looking at the command).
If no interrupt was pending, or the command wasn't completed, check
to see if we've set the private 'grace period' flag. If so, the
command really *is* dead, so report it as dead and complete it with
a CAM_CMD_TIMEOUT value.
If the grace period flag wasn't set, set it and issue a SYNCHRONIZE_ALL
Marker Request Queue entry and re-set the timeout for one second
from now (see Revision 1.45 isp.c notes for more on this) to give
the firmware a final chance to complete this command.
store a bitmask of whether we've set a value into ccb->ccb_h.status,
whether we're in the watchdog routine for this command now, whether
we've set a grace period for this command and whether this command is
actually done.
See comments of rev 1.45 of isp.c for more complete information.
output mailbox values we want to get back out of the chip once a mailbox
command is done. Add storage for the maximum number of output mailbox
registers to the softc.
Roll minor version number.
the handle (i.e., generation number), so we will now need a function that
will take a handle and return a flat index [ 0 .. maxhandles-1 ] for
auxillary routines that need an index to get at buddy store values
(like dma maps or xflist pointers).
Force alphas to prefer mem mapping as the default.
Basically, we have a pointer to a function which we can call which will
return us a pointer to firmware for the card we have. We call this function
(if it's non-NULL) with the address of our mdvec f/w pointer.
The way this works is that if ispfw (as a module or a static) is loaded,
it initializes the pointer in isp_pci, so we can call into to it to fetch
a pointer to a f/w set.
If ispfw is MOD_UNLOADed, it's retained a pointer to our mdvec f/w pointers,
which then get zeroed out so we don't have any references to data that's
now gone from kernel memory. Removing the f/w saves ~360KBytes.
Alas, there is no autounload mechanism that works for is here.
through, establish what our LUN width is. Unfortunately, we can't ask
the f/w. If we loaded the f/w, we'll now assume we have expanded LUNs
(SCCLUN for fibre channel, just plain 32 LUN for SCSI). If we didn't
load firmware, assume 8 LUNs for SCSI and 1 LUN for Fibre Channel. We
have to assume only one LUN for Fibre Channel because the LUN setting
in Request Queue entries is in different places whether we have SCCLUN
firmware or not, so the only LUN guaranteed to work for both is LUN 0.
Clean up the rest of isp.c so that ISP2100_SCCLUN defines aren't used-
instead use run time determinants based upon isp->isp_maxluns.
After starting firmware, delay 500us to give it a chance to get rolling.
Fix the interrupt service routine to check for both isr && sema being zero
before thinking this was a spurious interrupt. Following the manuals,
allow for both Mailbox as well as Queue Reponse type interrupts for regular
SCSI.
(we always support fabric now). Remove SCCLUN definition (we always
support SCCLUN now, if we load the f/w). Add typedef definition of an
external firmware fetch function.
What we'd like to know is whether or not we have a listener
upstream that really hasn't configured yet. If we do, then
we can give a more sensible reply here. If not, then we can
reject this out of hand.
Choices for what to send were
Not Ready, Unit Not Self-Configured Yet
(0x2,0x3e,0x00)
for the former and
Illegal Request, Logical Unit Not Supported
(0x5,0x25,0x00)
for the latter.
We used to decide whether there was at least one listener
based upon whether the black hole driver was configured.
However, recent config(8) changes have made this hard to do
at this time.
Actually, we didn't use the above quite yet, but were sure considering it.
changes: consider a new PDB entry different if Class 3 service parameter
roles change (!!!). Do some checking as we're getting a port database
that traps whether things change while we're doing so. Handle N-port
and F-ports correctly. Fix the fabric login loop to retain a login/binding
if things haven't changed (I mean, why logout a device only to log it back
in). No longer accept, after fabric logins, garbage if we can't get a PDB
entry that matches the device we've just logged into- if it doesn't, log
it out as it is very unlikely to still be what we thought it was. Get rid
of some of the debounce loops because we could get stuck there.
Apparently the f/w has finished the command, but somehow an interrupt is
being lost. So, we just plain wedge when booting alphas.
This is a general routine we've needed for a while.
where we can have targets (based on topology).
Much more importantly, make sure all mods to isp_sendmarker or |= so
we don't lose the marking of a bus that needs to have a marker sent for it.
require full logins after a LIP, which always led to loop resets, and
various other perturbations.
Update 2200 f/w from 2.01.00 release to 2.01.09 release.
seriously- only attempt to logout a previously logged in fabric device.
Fix a longstanding bug for aborting overtime commands- handle halves
have always been reversed.
Clean up some error messages to indicate channel number.
Approved:jkh
Andrew's problems with SCSI on some alphas- do not call isp_update
directly to update parameters- just mark them as being ready to
update for the next command- the system would just hang on a READ
CAPACITY for a drive. Really annoying because it wouldn't even timeout
(and it has a timeout) so either the SET PARAMETERS call was nuking
things or the f/w was really dropping the ball.
approved: jkh
Reviewed by: gallatin@freebsd.org
is gone as a define. We just don't support fast posting for anything less
than the 1240/1080/1280/12160 or Fibre Channel cards.
Put in support for CDB's larger than 12 bytes for parallel SCSI (up to 44
bytes are allowed).
Approved: jkh
for 1020/1X80/12160/2X00- for readability. Add in 12160 (Ultra3)
support- but not with PPR just yet. Fix and clarify fetching of
return parameter for getting firmware rev which for the 2200 contains
the connection topology (Private Loop (NL-port), N-port, FL-port,
F-port). Synthesize the connection topology for the 2100 which can
only be Private Loop or FL-port. Handle a couple of new async
mailbox commands which signify connection in Point-to-Point mode
(N-port or F-port) or indicate various toe stubbing getting to same.
Approved: jkh@freebsd.org
code gratefully borrowed from Patrick Stirling who did a lot of the
grunt work on this years ago. There are also some beginnings of
swizzle macros in case we go to a big endian machine. This is just
a first pass at this and is likely to change a bit over the next
Add in a very large amount of target mode support code- this is just
a first pass at this. It's a difficult thing because some of the code
can be in platform independent areas (see isp_target.?) but a lot has
to be in platform dependent areas because of not only the tight coupling
of received commands/events and the specific OS subsystem but because
the platform independent code has (deliberately) no event/wait mechanisms.
of where we could have seen the loop up at least once so it
makes sense. Change some stuff in ispscsicmd so we don't get
stuck there if the loop has never come up yet. Add in some
target mode support code.
of queue entries have to be at least 16 bits now! If we're running
a 2100 less than rev 5, turn off loop fairness (per Qlogic errata). Fix
typo in checking against 2200 F/W revision. Slightly fix/reorder fabric
login stuff. Change to usage of isp_getrqentry for code clarity. Add some
defensive dual bus assumptions. Various cleanups, etc...
Role the core version minor number. Change the arguments to the dma
setup function to use a u_int16_t for the output request loop pointer
(truly amazing that this hasn't blown up in anyones face so far). Do
some shuffling around of some items.
by Qlogic. The firmware is now also kept from compiling by default
unless some config options are set.
While we're at it, roll the 2200 f/w to 2.01.0. Still need to get the
1.17.26 2100 f/w working as it solves a lot of problems but it doesn't
want to work yet with this driver (:-)).
out of the PCI CLASS reg and store it in the softc. Use the getenv_quad
function to get a WWN override from the environment. Look for a config
value for same. Make slightly less lame the wwn seed construction.
have you is prototyped). Removed code versions in md struct- not used
any more. Allocate transfer dma maps and xflist stuff in mbxdmasetup based
upon isp->isp_maxcmds. Allow for multiple calls to mbxdmasetup (for
isp_reset cases).
file later. Do some pencil-sharpening types of minor changes. Change
how active commands are remembered (using new inline functions to get
handles, etc..). Now do a GET FIRMWARE STATUS after firing up the f/w as
outgoing mailbox 2 will tell you the f/w's notion of the max commands
that can be supported. Attempt to retrieve loop topology. Add in the
appropriate SWIZZLE/UNSWIZZLE macros calls (this is a no-op on Little
Endian machines but is needed for sparc (on other platforms)). Move
the temp port database we use to find out where things have moved to
after a LIP to the softc and off the kernel stack. Follow Qlogic's
hint and don't bother setting a tag for commands that don't have
this enabled (presumably the f/w will do it's own selection then).
Use an INT_PENDING macro to check for an interrupt. The call to
ISP_DMAFREE now just takes the handle- not the 'handle-1' which was
a layering violation. Use CFGPRINTF in a couple of places to make
things less chatty if not booting verbose, or CAMDEBUG compiles, etc..
where it defaults to one. Change simq width allocation to the max number
of commands supported by the HBA after f/w fires up- not the constant
MAXISPREQUEST value. Do some stylistic changes.
Add in null SWIZZLE definitions. Add in CFGPRINTF define. Change default
debug level to refer to an external isp_debug variable. Remove inline
functions as they're now in isp_inline.h and include that file.
the result queue length is never less than 64. Move (ick) temp port
database used for post-LIP merging off the kernel stack and put it
into the softc. Remove some target mode stuff which will come back
later in a different file. Change how the list of outstanding commands
are stored (now allocated at mailbox setup time to be just enough for
the max for a specific HBA which can vary). Keep a rotating seed of
the last index for this in the softc. Increase the count of active
commands from 10 to 16 bits.
isp_io_map, isp_no_fwload, isp_fwload, isp_no_nvram, isp_fcduplex
which are all bitmaps of isp instances that should or shouldn't
map memory space, I/O space, not load f/w, load f/w, ignore nvram,
not ignore nvarm, set full duplex mode. Also have an isp_seed value
that we can use to generate a pseudo seed for a synthetic WWN.
Other minor cosmetic cleanup. Add in support for the Qlogic ISP
2200. Very important change where we actually check now to see
whether we were successful in mapping request and response queues
(and fibre channel scratch space).
not having SCSI_ISP_SCCLUN config defined if we don't have f/w for
the 2200- it's resident firmware uses SCCLUN (65535 luns)). Change
the way the default LoopID is gathered (it's now a platform specific
define so that some attempt at a synthetic WWN can be made in case
NVRAM isn't readable).
Change initialization of options a bit- don't use ADISC. Set
FullDuplex mode if config options tells us to do so. Do not use
FULL_LOGIN after LIP- it's the right thing to do but it causes too
much loop disruption (Loop Resets). Sanity check some default
values. Redo construction of port and node WWNs based upon what we
have- if we have 2 in the top nibble, we can have distinct port
and node WWNs. Clean up some SCCLUN related code that we obviously
had never compiled (:-(). Audit commands coming int ispscsicmd and
don't throw commands at Fibre devices that do not have Class 3
service parameters TARGET ROLE defined.
Clean up f/w initialization a bit. Add Fabric support (or at least
the first blush of it). Whew - way too much to describe here.
Basically, after a LIP, hang out until we see a Loop Up or a Port
DataBase Change async event, then see if we're on a Fabric
(GET_PORT_NAME of FL_PORT_ID). If we are, try and scan the fabric
controller for fabric devices using the GetAllNext SNS subcommand.
As we find devices, announce them to the outer layer. Try and do
some guard code for broken (Brocade) SNS servers (that get stuck
in loops- gotta maybe do this a different way using the GP_ID3 cmd
instead). Then do a scan of the lower (local loop) ids using a
GET_PORT_NAME to see if the f/w has logged into anything at that
loop id. If so, then do a GET_PORT_DATABASE command. Do this scan
into a local database. At this point we can say the loop is 'Ready'.
After this, we merge our local loop port database with our stored
port database- in a as yet to be really fully exercised fashion we
try and follow the logic of something having moved around. The
first time we see something at a Loop ID, we fix it, for the purpose
of this system instance, at that Loop ID. If things shift around
so it ends up somewhere else, we still keep it at this Loop ID (our
'Target') but use the new (moved) Loop ID when we actually throw
commands at it. Check for insane cases of different Loop IDs both
claiming to have the same WWN- if that happens, invalidate both.
Notify the outer layer of devices that have arrived and devices
that have gone away. *Finally*, when this is done, search the
softc's database of Fabric devices and perform logout/login actions.
The Qlogic f/w maintains logout/login for all local loop devices.
We have to maintain logout/login for fabric devices- total PITA.
Expect to see this area undergo more change over time.
Change some fcp parameter structures such that we can get the portid
(24 bit value), get both node and port WWN, know whether we're on a fabric
or not, note whether we've ever seen the loop up, and note the current
state of the loop.
Replace the isp_pdb_t structure in fcparams with a reduced cost structure
that maintains a static relationship to 'Target', but can have the actual
loop ID used change (in case, post LIP, we discover things have moved
around). This also retains portid and node/port WWNs. This array gets
larger if we have fabric support compiled in.
Note special loop IDs that are invariate for this device- FL_PORT_ID
(0x7e) which tells us if there's a fabric controller present, FC_PORT_ID
and FC_SNS_ID (fabric controller port and fabric SNS server port). We don't
use the latter two for anything. IDs above FC_SNS_ID up through 255 are
available for mapping fabric devices to 'target' ids.
Add in a config define to set FC full duplex mode. Add in a define to
recognize the Qlogic 2200 boards. Add comments about ISPCTL commands.
Add and change some ISPASYNC enumes.
massive thwunking to include an XS_CHANNEL value. Some changes of how
parameters are reported to outer layers (including bus, e.g.). Yet more
stirring around in isp_mboxcmd to try and get it right. Decode of 1080/1240
NVRAM.
