load f/w images > 0x7fff words), set ISP_FW_ATTR_SCCLUN. We explicitly
don't believe we can find attributes if f/w is < 1.17.0, so we have to
set SCCLUN for the 1.15.37 f/w we're using manually- otherwise every
target will replicate itself across all 16 supported luns for non-SCCLUN
f/w.
Correctly set things up for 23XX and either fast posting or ZIO. The
23XX, it turns out, does not support RIO. If you put a non-zero value
in xfwoptions, this will disable fast posting. If you put ICBXOPT_ZIO
in xfwoptions, then the 23XX will do interrupt delays but post to the
response queue- apparently QLogic *now* believes that reading multiple
handles from registers is less of a win than writing (and delaying)
multiple 64 byte responses to the response queue.
At the end of taking a a good f/w crash dump, send the ISPASYNC_FW_DUMPED
event to the outer layers (who can then do things like wake a user
daemon to *fetch* the crash image, etc.).
turns out that there's something of a hole in our new fabric name
server stuff. We ask the name server for entities that have
registered as a specific type. That type is FC-SCSI. If the entity
hasn't performed a REGISTER FC4 TYPES, the fabric nameserver won't
return it.
This brings this driver to a bit of a fork in the road as to what
the right thing to do is. For servicing the needs of accessing
FC-SCSI devices, this method is fine, and to be preferred. It is
extremely unlikely we're interested in fabric devices that *don't*
register correctly. If I ever get around to adding an FC-IP stack,
then asking for devices that have registers as FC-IP types is also
the right thing to do.
So- asking the fabric nameserver for a specific type is fine, *as
long as you are only interested in specific types*. If, on the other
hand, you want to create (as for management tool support) a picture
of everything on the fabric, this is *not* so fine. There are a
large class of FC-SCSI *initiators* who *don't* correctly register,
so we never will *see* them.
Is this a problem? Yes, but only a little one. If we want to do such
management tool support, we should probably run a *different* fabric
nameserver query algorithm. Better yet, we should talk to the management
nameserver in Brocade switches instead of the standard FC-GS-2 fabric
nameserver (which can be unwieldy).
Other changes: if we've overrrides marked, don't set some default
values from reading NVRAM. This allows us to override things like
EXEC throttle without having to ignore NVRAM entirely.
MFC after: 1 week
CAM_QUIRK_HILUN devices we loop thru 32bits of lun. Oops.
Switch to using USEC_DELAY rather than USEC_SLEEP at isp_reset time.
Try to paper around a defect in clients that don't correctly registers
themeselves with the fabric nameserver.
Minor updates for Mirapoint support- they still use code that is not
HANDLE_LOOPSTATE_IN_OUTER_LAYERS, and, surprise surprise, this old
stuff had some bugs in it.
Clean up some target mode stuff.
MFC after: 1 week
is not set in the scsi completion status, or if the residual is clearly
nonsense, then this was a command that suffered the loss of one or more
FC frames in the middle of the exchange.
Set HBA_BOTCH and hope it will get retried. It's the only thing we can do.
MFC after: 1 day
lun address modifier of sorts. Only an HP XP-512 seems to have cared.
Fix a few misplaced pointers for the new fabric goop, which has been
demonstrated to work on newer Brocades and McData switches now.
Put in commented out code which would run GFF_ID if the QLogic f/w
allowed it.
Don't whine about not being able to find a handle for a command if it
was a command aborted (by us).
Grumble. I've seen better documented architectures out of Redmond.
Redo fabric evaluation to not use GET ALL NEXT (GA_NXT). Switches seem
to be trying to wriggle out of supporting this well. Instead, use
GID_FT to get a list of Port IDs and then use GPN_ID/GNN_ID to find the
port and node wwn. This should make working on fabrics a bit cleaner and
more stable.
This also caused some cleanup of SNS subcommand canonicalization so that
we can actually check for FS_ACC and FS_RJT, and if we get an FS_RJT,
print out the reason and explanation codes.
We'll keep the old GA_NXT method around if people want to uncomment a
controlling definition in ispvar.h.
This also had us clean up ISPASYNC_FABRICDEV to use a local lportdb argument
and to have the caller explicitly say that a device is at the end of the
fabric list.
MFC after: 1 week
soon because it's just getting harder and harder to find switches
that correctly implement the GET ALL NEXT subcommands for the SNS
protocol.
Latch up result out pointer and set a busy flag when we're looking
at the response queue. This allows for a cleaner way to make sure
we don't get multiple CPUs trying to read the same response queue
entries.
Change how isp_handle_other_response returns values (clarity).
Make PORT UNAVAILABLE the same as PORT LOGOUT (force a LIP).
Do some formatting changes.
MFC after: 0 days
it worked- but I ran into a case with a 2204 where commands were being lost
right and left. Best be safe.
For target mode, or things called if we call isp_handle_other response- note
that we might have dropped locks by changing the output pointer so we bail
from the loop. It's the responsibility of the entity dropping the lock to
make sure that we let the f/w know we've read thus far into the response
queue (else we begin processing the same entries again- blech!).
MFC after: 1 day
OUT status. We are, apparently, required to force the f/w to log back in
if we want to try and talk to that disk again. This means either issuing
a LOGIN LOCAL LOOP PORT mailbox command, or by issuing a LIP. I've elected
to issue a LIP because this has a better chance of waking up the disk which
clearly just crashed and burned.
These should not occur at all. If they do, they should be darned rare.
MFC after: 1 week
If you want QLogic to look at a potential f/w problem for FC cards, you really
have to provide them info in the format they expect. This involves dumping
a lot of hardware registers (> 300 16 bit registers) and a lot of SRAM
(> 128KB minimum). Thus all of this code is #ifdef protected which will
become an option so that the memory allocation of where to dump the crash
image is pretty expensive. It's worth it if you have a reproducible problem
because they have some tools that can tell them, given the f/w version,
the precise state of everything.
MFC after: 1 week
disable MWI on 2300
based on function code, set an 'isp_port' for the 2312- it's a
separate instance, but the NVRAM is shared, and the second port's
NVRAM is at offset 256.
+ Enable RIO operation for LVD SCSI cards. This makes a *big* difference
as even under reasonable load we get batched completions of about 30
commands at a time on, say, an ISP1080.
+ Do 'continuation' mailbox commands- this allows us to specify a work
area within the softc and 'continue' repeated mailbox commands. This is
more or less on an ad hoc basis and is currently only used for firmware
loading (which f/w now loads substantially faster becuase the calling
thread is only woken when all the f/w words are loaded- not for each
one of the 40000 f/w words that gets loaded).
+ If we're about to return from isp_intr with a 'bogus interrupt' indication,
and we're not a 23XX card, check to see whether the semaphore register is
currently *2* (not *1* as it should be) and whether there's an async completion
sitting in outgoing mailbox0. This seems to capture cases of lost fast posting
and RIO interrupts that the 12160 && 1080 have been known to pump out under
extreme load (extreme, as in > 250 active commands).
+ FC_SCRATCH_ACQUIRE/FC_SCRATCH_RELEASE macros.
+ Endian correct swizzle/unswizzle of an ATIO2 that has a WWPN in it.
MFC after: 1 week
firmware to delay completion of commands so that it can attempt to batch
a bunch of completions at once- either returning 16 bit handles in mailbox
registers, or in a resposne queue entry that has a whole wad of 16 bit handles.
Distinguish between 2300 and 2312 chipsets- if only because the revisions
on the chips have different meanings.
Add more instrumentation plus ISP_GET_STATS and ISP_CLR_STATS ioctls.
Run up the maximum number of response queue entities we'll look at
per interrupt.
If we haven't set HBA role yet, always return success from isp_fc_runstate.
MFC after: 2 weeks
a GetAllNext response. Otherwise, we won't unswizzle
it correctly. This was found on linux/PPC.
This mandated creating another inline: isp_get_gan_response.
the response queue. Instead of the ad hoc ISP_SWIZZLE_REQUEST, we now have
a complete set of inline functions in isp_inline.h. Each platform is
responsible for providing just one of a set of ISP_IOX_{GET,PUT}{8,16,32}
macros.
The reason this needs to be done is that we need to have a single set of
functions that will work correctly on multiple architectures for both little
and big endian machines. It also needs to work correctly in the case that
we have the request or response queues in memory that has to be treated
specially (e.g., have ddi_dma_sync called on it for Solaris after we update
it or before we read from it). It also has to handle the SBus cards (for
platforms that have them) which, while on a Big Endian machine, do *not*
require *most* of the request/response queue entry fields to be swizzled
or unswizzled.
One thing that falls out of this is that we no longer build requests in the
request queue itself. Instead, we build the request locally (e.g., on the
stack) and then as part of the swizzling operation, copy it to the request
queue entry we've allocated. I thought long and hard about whether this was
too expensive a change to make as it in a lot of cases requires an extra
copy. On balance, the flexbility is worth it. With any luck, the entry that
we build locally stays in a processor writeback cache (after all, it's only
64 bytes) so that the cost of actually flushing it to the memory area that is
the shared queue with the PCI device is not all that expensive. We may examine
this again and try to get clever in the future to try and avoid copies.
Another change that falls out of this is that MEMORYBARRIER should be taken
a lot more seriously. The macro ISP_ADD_REQUEST does a MEMORYBARRIER on the
entry being added. But there had been many other places this had been missing.
It's now very important that it be done.
Additional changes:
Fix a longstanding buglet of sorts. When we get an entry via isp_getrqentry,
the iptr value that gets returned is the value we intend to eventually plug
into the ISP registers as the entry *one past* the last one we've written-
*not* the current entry we're updating. All along we've been calling sync
functions on the wrong index value. Argh. The 'fix' here is to rename all
'iptr' variables as 'nxti' to remember that this is the 'next' pointer-
not the current pointer.
Devote a single bit to mboxbsy- and set aside bits for output mbox registers
that we need to pick up- we can have at least one command which does not
have any defined output registers (MBOX_EXECUTE_FIRMWARE).
MFC after: 2 weeks
If we get a completion status of RQCS_QUEUE_FULL, it means
that the internal queues are full. Other QLogic boards set
the QFULL SCSI status. But *nooooooooooo*, not the 2300.
MFC after: 1 day
once so there isn't a window with the ones for the 23XX cards being wrong.
When being verbose, print out some more FC NVRAM values (like framesize).
MFC after: 1 week
luns) firmware for the Fibre Channel cards.
We used to assume that if we didn't download firmware, we couldn't know
what the firmware capability with respect to SCCLUNs is- and it's important
because the lun field changes in the request queue entry based upon which
firmware it is.
At any rate, we *do* get back firmware attributes in mailbox register 6
when we do ABOUT FIRMWARE for all 2200/2300 cards- and for 2100 cards
with at least 1.17.0 firmware. So- we now assume non-SCCLUN behaviour
for 2100 cards with firmware < 1.17.0- and we check the firmware attributes
for other cards (loaded firmware or not).
This also allows us to get rid of the crappy test of isp_maxluns > 16-
we simply can check firmware attributes for SCCLUN behaviour.
This required an 'oops' fix to the outgoing mailbox count field for
ABOUT FIRMWARE for FC cards.
Also- while here, hardwire firmware revisions for loaded code for SBus
cards. Apparently the 1.35 or 1.37 f/w we've been loading into isp1000
just doesn't report firmware revisions out to mailbox regs 1, 2 and 3
like everyone else. Grumble. Not that this fix hardly matters for FreeBSD.
MFC after: 4 weeks
some reworking (and consequent cleanup) of the interrupt service code.
Also begin to start a cleanup of target mode support that will (eventually)
not require more inforamtion routed with the ATIO to come back with the
CTIO other than tag.
MFC after: 4 weeks
that do not have valid NVRAM. In particular, we were leaving
a retry count set (to retry selection timeouts) when thats
not really what we want. Do some constant string additions
so that LOGDEBUG0 info is useful across all cards.
MFC after: 2 weeks
either what's in NVRAM or what the safe defaults would be if we lack NVRAM.
Then we rename cur_XXXX to actv_XXXX (these are the currently active settings)
and the dev_XXX settings to goal_XXXX (these are the settings which we want
cur_XXXX to converge to).
We originally had it such that if the connection topology was FL-loop
(public loop), we never looked at any local loop addresses. The reason
for not doing that was fear or concern that we'd see the same local
loop disks reflected from the name server and we'd attach them twice.
However, when I recently hooked up a JBOD and a system to an ANCOR SA-8
switch, the disks did *not* show up on the fabric. So at least the
ANCOR is screening those disks from appearing on the fabric. Now, it's
possible this is a 'feature' of the ANCOR. When I get a chance, I'll
check the Brocade (it's hard to do this on a low budget).
In any case, if they *do* also show up on the fabric, we should
simply elect to not log into them because we already have an
entry for the local loop. There is relatively unexercised code
just for this case.
MFC after: 2 weeks
part of the PCI block for the 2300- not software convention usage
of the mailbox registers- so we macrosize in/out pointer usage.
Only report that a LIP destroyed commands if it actually destroyed
commands. Get the chan/tgt/lun order correct. Fix a longstanding
stupid bug that caused us to try and issue a command with a tag on
Channel B because we were checking the tagged capability for the
target against Channel A.
A firmware crash is now vectored out to platform specific code
as an async event.
Some minor formatting tweaks.
commands that complete (with no apparent error) after
we receive a LIP. This has been observed mostly on
Local Loop topologies. To be safe, let's just mark
all active commands as dead if we get a LIP and we're
on a private or public loop.
MFC after: 4 weeks
----
Make a device for each ISP- really usable only with devfs and add an ioctl
entry point (this can be used to (re)set debug levels, reset the HBA,
rescan the fabric, issue lips, etc).
----
Add in a kernel thread for Fibre Channel cards. The purpose of this
thread is to be woken up to clean up after Fibre Channel events
block things. Basically, any FC event that casts doubt on the
location or identify of FC devices blocks the queues. When, and
if, we get the PORT DATABASE CHANGED or NAME SERVER DATABASE CHANGED
async event, we activate the kthread which will then, in full thread
context, re-evaluate the local loop and/or the fabric. When it's
satisfied that things are stable, it can then release the blocked
queues and let commands flow again.
The prior mechanism was a lazy evaluation. That is, the next command
to come down the pipe after change events would pay the full price
for re-evaluation. And if this was done off of a softcall, it really
could hang up the system.
These changes brings the FreeBSD port more in line with the Solaris,
Linux and NetBSD ports. It also, more importantly, gets us being
more proactive about topology changes which could then be reflected
upwards to CAM so that the periph driver can be informed sooner
rather than later when things arrive or depart.
---
Add in the (correct) usage of locking macros- we now have lock transition
macros which allow us to transition from holding the CAM lock (Giant)
and grabbing the softc lock and vice versa. Switch over to having this
HBA do real locking. Some folks claim this won't be a win. They're right.
But you have to start somewhere, and this will begin to teach us how
to DTRT for HBAs, etc.
--
Start putting in prototype 2300 support. Add back in LIP
and Loop Reset as async events that each platform will handle.
Add in another int_bogus instrumentation point.
Do some more substantial target mode cleanups.
MFC after: 8 weeks
Redo establishment of default SCSI parameters whether or not
we've been compiled for target mode. Unfortunately, the Qlogic
f/w is confused so that if we set all targets to be 'safe' (i.e.,
narrow/async), it will also then report narrow, async if we're
contacted in target mode from that target (acting in initiator
role). D'oh!
Fix ISPCTL_TOGGLE_TMODE to correctly enable the right channel for
dual channel cards. Add some more opcodes. Fix a stupid NULL
pointer bug.
Solaris (which, for reasons unknown to me, chokes on u_int16_t
as a typedef of unsigned short if used in a transitional (mixed K&R
and ANSI) way), we'll go the extra mile and fully ANSIfy things.
This is a pretty invasive change, but there are three good
reasons to do this:
1. We'll never have > 16 bits of handle.
2. We can (eventually) enable the RIO (Reduced Interrupt Operation)
bits which return multiple completing 16 bit handles in mailbox
registers.
3. The !)$*)$*~)@$*~)$* Qlogic target mode for parallel SCSI spec
changed such that at_reserved (which was 32 bits) was split into
two pieces- and one of which was a 16 bit handle id that functions
like the at_rxid for Fibre Channel (a tag for the f/w to correlate
CTIOs with a particular command). Since we had to muck with that
and this changed the whole handler architecture, we might as well...
Propagate new at_handle on through int ct_fwhandle. Follow
implications of changing to 16 bit handles.
These above changes at least get Qlogic 1040 cards working in target
mode again. 1080/12160 cards don't work yet.
In isp.c:
Prepare for doing all loop management in outer layers.
for the ICB firmware options meant- *I* had taken it to
mean that if you set it, Node Name would be ignored and
derived from Port Name. Actually, it meant the opposite.
As a consequence- change ICBOPT_USE_PORTNAME to the
define ICBOPT_BOTH_WWNS- makes more sense.
Fix wrong input bitmap for MBOX_DUMP_RAM command. Call
ISP_DUMPREGS if we get a f/w crash. Add ISPCTL_RUN_MBOXCMD
control command (so outer layers can run a mailbox command
directly) and add a ISPASYNC_UNHANDLED_RESPONSE hook so
outer layers can understand response queue entries we
might not know about.
Remove ISP2100_FABRIC defines- we always handle fabric now. Insert
isp_getmap helper function (for getting Loop Position map). Make
sure we (for our own benefit) mark req_state_flags with RQSF_GOT_SENSE
for Fibre Channel if we got sense data- the !*$)!*$)~*$)*$ Qlogic
f/w doesn't do so. Add ISPCTL_SCAN_FABRIC, ISPCTL_SCAN_LOOP, ISPCTL_SEND_LIP,
and ISPCTL_GET_POSMAP isp_control functions. Correctly send async notifications
upstream for changes in the name server, changes in the port database, and
f/w crashes. Correctly set topology when we get a ASYNC_PTPMODE event.
Major stuff:
Quite massively redo how we handle Loop events- we've now added several
intermediate states between LOOP_PDB_RCVD and LOOP_READY. This allows us
a lot finer control about how we scan fabric, whether we go further
than scanning fabric, how we look at the local loop, and whether we
merge entries at the level or not. This is the next to last step for
moving managing loop state out of the core module entirely (whereupon
loop && fabric events will simply freeze the command queue and a thread
will run to figure out what's changed and *it* will re-enable the queu).
This fine amount of control also gets us closer to having an external
policy engine decide which fabric devices we really want to log into.
Module) and FBM (Fibre Buffer Modules). Also remember to clear the
semaphore registers. Tell the RISC processor to not halt on FPM
parity errors.
Throw out the ISP_CFG_NOINIT silliness and instead go to the use of
adapter 'roles' to see whether one completes initialization or not
(mostly for Fibre Channel). The ultimate intent, btw, of all of this
is to have a warm standby adapter for failover reasons. Because
we do roles now, setting of Target Capable Class 3 service parameters
in the ICB for the 2x00 cards reflects from role. Also, in isp_start,
if we're not supporting an initiator role, we bounce outgoing commands
with a Selection Timeout error. Also clean out the TOGGLE_TMODE
goop for FC- there is no toggling of target mode like there is
for parallel SCSI cards.
Do more cleanup with respect to using target ids 0..125 in F-port
topologies. Also keep track of things which *were* fabric devices
so that when you rescan the fabric you can notify the outer layers
when fabric devices go away.
Only force a LOGOUT for fabric devices if they're still logged in
(i.e., you cat their Port Database entry. Clean up the Get All Next
scanning.
Finally, use a new tag in the softc to store the opcode for the
last mailbox command used so we can report which opcode timed
out.
that require us to register our FC4 types of interest. Allow ourselves, in
F-port topologies, to start logging in fabric devices in the target 0..125
range. Change ISPASYNC_PDB_CHANGED (misnamed) to ISPASYNC_LOGGED_INOUT.
Fix (*SMACK*) again some default WWN stuff. This is *really* hard to get
right across all the range of platforms.
WWNs correctly (Again!) - this time for the case that we're not going
to fully init the adapter if isp_init is called (with ISP_CFG_NOINIT
set in options). The pupose for this is to bring the adapter up to
almost ready to go, get info out of NVRAM, but to not start it up- leaving
it until later to actually start things up if wanted (and possibly with
different roles selected).
(specifically, how many entries we've looked at so far). Maintain
interrupt instrumentation. Use USEC_SLEEP instead of USEC_DELAY in
a number of places (this allows us to drop locks and sleep instead
of spin). Track changes to configuration options for topology preference.
Fix botched order of printout for Channel, Target, Lun.
changes is that there's now a Solaris port of this driver, so some things
in the core version had to change (not much, but some).
In order, from the top.....:
A lot of error strings are gathered in one place at the head of the file.
This caused me to rewrite them to look consistent (with respect to
things like 'Port 0x%' and 'Target %d' and 'Loop ID 0x%x'.
The major mailbox function, isp_mboxcmd, now takes a third argument,
which is a mask that selectively says whether mailbox command failures
will be logged. This will substantially reduce a lot of spurious noise
from the driver.
At the first run through isp_reset we used to try and get the current
running firmware's revision by issuing a mailbox command. This would
invariably fail on alpha's with anything but a Qlogic 1040 since SRM
doesn't *start* the f/w on these cards. Instead, we now see whether we're
sitting ROM state before trying to get a running BIOS loaded f/w version.
All CFGPRINTF/PRINTF/IDPRINTF macros have been replaced with calls to
isp_prt. There are seperate print levels that can be independently
set (see ispvar.h), which include debugging, etc.
All SYS_DELAY macros are now USEC_DELAY macros. RQUEST_QUEUE_LEN and
RESULT_QUEUE_LEN now take ispsoftc as a parameter- the Fibre Channel
cards and the Ultra2/Ultra3 cards can have 16 bit request queue entry
indices, so we can make a 1024 entry index for them instead of the
256 entries we've had until now.
A major change it to fix isp_fclink_test to actually only wait the
delay of time specified in the microsecond argument being passed.
The problem has always been that a call to isp_mboxcmd to get he
current firmware state takes an unknown (sometimes long) amount of
time- this is if the firmware is busy doing PLOGIs while we ask
it what's up. So, up until now, the usdelay argument has been
a joke. The net effect has been that if you boot without being plugged
into a good loop or into a switch, you hang. Massively annonying, and
hard to fix because the actual time delta was impossible to know
from just guessing. Now, using the new GET_NANOTIME macros, a precise
and measured amount of USEC_DELAY calls are done so that only the
specified usecdelay is allowed to pass. This means that if the initial
startup of the firmware if followed by a call from isp_freebsd.c:isp_attach
to isp_control(isp, ISP_FCLINK_TEST, &tdelay) where tdelay is 2 * 1000000,
no more than two seconds will actually elapse before we leave concluding
that the cable is unhooked. Jeez. About time....
Change the ispscsicmd entry point to isp_start, and the XS_CMD_DONE
macro to a call to the platform supplied isp_done (sane naming).
Limit our size of request queue completions we'll look at at interrupt
time. Since we've increased the size of the Request Queue (and the
size of the Response Queue proportionally), let's not create an
interrupt stack overflow by having to keep a max completion list
(forw links are not an option because this is common code with
some platforms that don't have link space in their XS_T structures).
A limit of 32 is not unreasonable- I doubt there'd be even this many
request queue completions at a time- remember, most boards now use
fast posting for normal command completion instead of filling out
response queue entries.
In the isp_mboxcmd cleanup, also create an array of command
names so that "ABOUT FIRMWARE" can be printed instead of "CMD #8".
Remove the isp_lostcmd function- it's been deprecated for a while.
Remove isp_dumpregs- the ISP_DUMPREGS goes to the specific bus
register dump fucntion.
Various other cleanups.
define). Fix stupidity wrt checking whether we've gone to
LOOP_PDB_RCVD loopstate- it's okay to be greater than this state.
D'oh! Protect calls to isp_pdb_sync and isp_fclink_state with IS_FC
macros.
Completely redo mailbox command routine (in preparation to make this
possibly wait rather than poll for completion).
Make a major attempt to solve the 'lost interrupt' problem
1. Problem
The Qlogic cards would appear to 'lose' interrupts, i.e., a legitimate
regular SCSI command placed on the request queue would never complete
and the watchdog routine in the driver would eventually wakeup and
catch it. This would typically only happen on Alphas, although a
couple folks with 700MHz Intel platforms have also seen this.
For a long time I thought it was a foulup with f/w negotiations of
SYNC and/or WIDE as it always seemed to happen right after the
platform it was running on had done a SET TARGET PARAMETERS mailbox
command to (re)enable sync && wide (after initially forcing
ASYNC/NARROW at startup). However, occasionally, the same thing
would also occur for the Fibre Channel cards as well (which, ahem,
have no SET TARGET PARAMETERS for transfer mode).
After finally putting in a better set of watchdog routines for the
platforms for this driver, it seemed to be the case that the command
in question (usually a READ CAPACITY) just had up and died- the
watchdog routine would catch it after ~10 seconds. For some platforms
(NetBSD/OpenBSD)- an ABORT COMMAND mailbox command was sent (which
would always fail- indicating that the f/w denied knowledge of this
command, i.e., the f/w thought it was a done command). In any case,
retrying the command worked. But this whole problem needed to be
really fixed.
2. A False Step That Went in The Right Direction
The mailbox code was completely rewritten to no longer try and grab
the mailbox semaphore register and to try and 'by hand' complete
async fast posting completions. It was also rewritten to now have
separate in && out bitpatterns for registers to load to start and
retrieve to complete. This means that isp_intr now handles mailbox
completions.
This substantially simplifies the mailbox handling code, and carries
things 90% toward getting this to be a non-polled routine for this
driver.
This did not solve the problem, though.
3. Register Debouncing
I saw some comments in some errata sheets and some notes in a Qlogic
produced Linux driver (for the Qlogic 2100) that seemed to indicate
that debouncing of reads of the mailbox registers might be needed,
so I added this. This did not affect the problem. In fact, it made
the problem worse for non-2100 cards.
5. Interrupt masking/unmasking
The driver *used* to do a substantial amount of masking/unmasking
of the interrupt control register. This was done to make sure that
the core common code could just assume it would never get pre-empted.
This apparently substantially contributed to the lost interrupt
problem. The rewrite of the ICR (Interrupt Control Register),
which is a separate register from the ISR (Interrupt Status Register)
should not have caused any change to interrupt assertions pending.
The manual does not state that it will, and the register layout
seems to imply that the ICR is just an active route gate. We only
enable PCI Interrupts and RISC Interrupts- this should mean that
when the f/w asserts a RISC interrupt and (and the ICR allows RISC
Interrupts) and we have PCI Interrupts enabled, we should get a
PCI interrupt. Apparently this is a latch- not a signal route.
Removing this got rid of *most* but not all, lost interrupts.
5. Watchdog Smartening
I made sure that the watchdog routine would catch cases where the
Qlogic's ISR showed an interrupt assertion. The watchdog routine
now calls the interrupt service routine if it sees this. Some
additional internal state flags were added so that the watchdog
routine could then know whether the command it was in the middle
of burying (because we had time it out) was in fact completed by
the interrupt service routine.
6. Occasional Constipation Of Commands..
In running some very strenous high IOPs tests (generating about
11000 interrupts/second across one Qlogic 1040, one Qlogic 1080
and one Qlogic 2200 on an Alpha PC164), I found that I would get
occasional but regular 'watchdog timeouts' on both the 1080 and
the 2100 cards. This is under FreeBSD, and the watchdog timeout
routine just marks the command in error and retries it.
Invariably, right after this 'watchdog timeout' error, I'd get a
command completion for the command that I had thought timed out.
That is, I'd get a command completion, but the handle returned by
the firmware mapped to no current command. The frequency of this
problem is low under such a load- it would usually take an 30
minutes per 'lost' interrupt.
I doubled the timeout for commands to see if it just was an edge
case of waiting too short a period. This has no effect.
I gathered and printed out microtimes for the watchdog completed
command and the completion that couldn't find a command- it was
always the case that the order of occurrence was "timeout, completion"
separated by a time on the order of 100 to 150 ms.
This caused me to consider 'firmware constipation' as to be a
possible culprit. That is, resubmission of a command to the device
that had suffered a watchdog timeout seemed to cause the presumed
dead command to show back up.
I added code in the watchdog routine that, when first entered for
the command, marks the command with a flag, reissues a local timeout
call for one second later, but also then issues a MARKER Request
Queue entry to the Qlogic f/w. A MARKER entry is used typically
after a Bus Reset to cause the f/w to get synchronized with respect
to either a Bus, a Nexus or a Target.
Since I've added this code, I always now see the occasional watchdog
timeout, but the command that was about to be terminated always
now seems to be completed after the MARKER entry is issued (and
before the timeout extension fires, which would come back and
*really* terminate the command).
through, establish what our LUN width is. Unfortunately, we can't ask
the f/w. If we loaded the f/w, we'll now assume we have expanded LUNs
(SCCLUN for fibre channel, just plain 32 LUN for SCSI). If we didn't
load firmware, assume 8 LUNs for SCSI and 1 LUN for Fibre Channel. We
have to assume only one LUN for Fibre Channel because the LUN setting
in Request Queue entries is in different places whether we have SCCLUN
firmware or not, so the only LUN guaranteed to work for both is LUN 0.
Clean up the rest of isp.c so that ISP2100_SCCLUN defines aren't used-
instead use run time determinants based upon isp->isp_maxluns.
After starting firmware, delay 500us to give it a chance to get rolling.
Fix the interrupt service routine to check for both isr && sema being zero
before thinking this was a spurious interrupt. Following the manuals,
allow for both Mailbox as well as Queue Reponse type interrupts for regular
SCSI.
changes: consider a new PDB entry different if Class 3 service parameter
roles change (!!!). Do some checking as we're getting a port database
that traps whether things change while we're doing so. Handle N-port
and F-ports correctly. Fix the fabric login loop to retain a login/binding
if things haven't changed (I mean, why logout a device only to log it back
in). No longer accept, after fabric logins, garbage if we can't get a PDB
entry that matches the device we've just logged into- if it doesn't, log
it out as it is very unlikely to still be what we thought it was. Get rid
of some of the debounce loops because we could get stuck there.
where we can have targets (based on topology).
Much more importantly, make sure all mods to isp_sendmarker or |= so
we don't lose the marking of a bus that needs to have a marker sent for it.
seriously- only attempt to logout a previously logged in fabric device.
Fix a longstanding bug for aborting overtime commands- handle halves
have always been reversed.
Clean up some error messages to indicate channel number.
Approved:jkh
is gone as a define. We just don't support fast posting for anything less
than the 1240/1080/1280/12160 or Fibre Channel cards.
Put in support for CDB's larger than 12 bytes for parallel SCSI (up to 44
bytes are allowed).
Approved: jkh
for 1020/1X80/12160/2X00- for readability. Add in 12160 (Ultra3)
support- but not with PPR just yet. Fix and clarify fetching of
return parameter for getting firmware rev which for the 2200 contains
the connection topology (Private Loop (NL-port), N-port, FL-port,
F-port). Synthesize the connection topology for the 2100 which can
only be Private Loop or FL-port. Handle a couple of new async
mailbox commands which signify connection in Point-to-Point mode
(N-port or F-port) or indicate various toe stubbing getting to same.
Approved: jkh@freebsd.org
of where we could have seen the loop up at least once so it
makes sense. Change some stuff in ispscsicmd so we don't get
stuck there if the loop has never come up yet. Add in some
target mode support code.
of queue entries have to be at least 16 bits now! If we're running
a 2100 less than rev 5, turn off loop fairness (per Qlogic errata). Fix
typo in checking against 2200 F/W revision. Slightly fix/reorder fabric
login stuff. Change to usage of isp_getrqentry for code clarity. Add some
defensive dual bus assumptions. Various cleanups, etc...
file later. Do some pencil-sharpening types of minor changes. Change
how active commands are remembered (using new inline functions to get
handles, etc..). Now do a GET FIRMWARE STATUS after firing up the f/w as
outgoing mailbox 2 will tell you the f/w's notion of the max commands
that can be supported. Attempt to retrieve loop topology. Add in the
appropriate SWIZZLE/UNSWIZZLE macros calls (this is a no-op on Little
Endian machines but is needed for sparc (on other platforms)). Move
the temp port database we use to find out where things have moved to
after a LIP to the softc and off the kernel stack. Follow Qlogic's
hint and don't bother setting a tag for commands that don't have
this enabled (presumably the f/w will do it's own selection then).
Use an INT_PENDING macro to check for an interrupt. The call to
ISP_DMAFREE now just takes the handle- not the 'handle-1' which was
a layering violation. Use CFGPRINTF in a couple of places to make
things less chatty if not booting verbose, or CAMDEBUG compiles, etc..
not having SCSI_ISP_SCCLUN config defined if we don't have f/w for
the 2200- it's resident firmware uses SCCLUN (65535 luns)). Change
the way the default LoopID is gathered (it's now a platform specific
define so that some attempt at a synthetic WWN can be made in case
NVRAM isn't readable).
Change initialization of options a bit- don't use ADISC. Set
FullDuplex mode if config options tells us to do so. Do not use
FULL_LOGIN after LIP- it's the right thing to do but it causes too
much loop disruption (Loop Resets). Sanity check some default
values. Redo construction of port and node WWNs based upon what we
have- if we have 2 in the top nibble, we can have distinct port
and node WWNs. Clean up some SCCLUN related code that we obviously
had never compiled (:-(). Audit commands coming int ispscsicmd and
don't throw commands at Fibre devices that do not have Class 3
service parameters TARGET ROLE defined.
Clean up f/w initialization a bit. Add Fabric support (or at least
the first blush of it). Whew - way too much to describe here.
Basically, after a LIP, hang out until we see a Loop Up or a Port
DataBase Change async event, then see if we're on a Fabric
(GET_PORT_NAME of FL_PORT_ID). If we are, try and scan the fabric
controller for fabric devices using the GetAllNext SNS subcommand.
As we find devices, announce them to the outer layer. Try and do
some guard code for broken (Brocade) SNS servers (that get stuck
in loops- gotta maybe do this a different way using the GP_ID3 cmd
instead). Then do a scan of the lower (local loop) ids using a
GET_PORT_NAME to see if the f/w has logged into anything at that
loop id. If so, then do a GET_PORT_DATABASE command. Do this scan
into a local database. At this point we can say the loop is 'Ready'.
After this, we merge our local loop port database with our stored
port database- in a as yet to be really fully exercised fashion we
try and follow the logic of something having moved around. The
first time we see something at a Loop ID, we fix it, for the purpose
of this system instance, at that Loop ID. If things shift around
so it ends up somewhere else, we still keep it at this Loop ID (our
'Target') but use the new (moved) Loop ID when we actually throw
commands at it. Check for insane cases of different Loop IDs both
claiming to have the same WWN- if that happens, invalidate both.
Notify the outer layer of devices that have arrived and devices
that have gone away. *Finally*, when this is done, search the
softc's database of Fabric devices and perform logout/login actions.
The Qlogic f/w maintains logout/login for all local loop devices.
We have to maintain logout/login for fabric devices- total PITA.
Expect to see this area undergo more change over time.
massive thwunking to include an XS_CHANNEL value. Some changes of how
parameters are reported to outer layers (including bus, e.g.). Yet more
stirring around in isp_mboxcmd to try and get it right. Decode of 1080/1240
NVRAM.
board versions with no BIOS. Separate mailbox interrupts from
IOCB interrupts. Read OUTMAILBOX5 while RISC_INT is active- not
after you clear it (potential race condition). Clear out older broken
BIG_ENDIAN goop. Don't negotiate narrow/async for LVD busses at startup
if already in LVD mode. Note usage of presumptive 1040C revision. For
all the LIP, PDB Changed, Loop UP/DOWN async events, mark fw state
as unknown as well as marking the need to do a getpdb on targets- after
a LIP for certain the f/w has to do PRLI/PLOGI for all targets again
and marking f/w state as unknown gives us a fighting chance to (start
to) hold up for that to complete.
settings you've just sent them and return random values if you follow
the set by a get. This causes problems when you latter run a Tag-enabled
command when you've command tagged mode off.
thwank in register layout goop). A different mboxcmd approach. Some PDB change
infrastructure. Some better management of loopdown/loopup events (keep them
distinct from resource starvation for simq freeze/unfreeze actions).
Use fast memory timing NVRAM parameter. Clean up and fix establishment
of default target parameters. Don't use NVRAM if are flagged as not to
do so (I had a busted NVRAM setup which I couldn't edit that enabled SYNC
mode but disabled disconnect/reconnect and wide!!). Fix delays after
resets. BUS resets not done in isp_init anymore- relegated to OS
specific outer layers. Fix a buglet where you can get in a loop for
a NULL xs in the completion list in isp_intr. Add in some defines that
can disable fast posting. Add in code for Loop Up/Loop Down events that
call into the outer layers as to what to do.
the startup code. Implement a call to outer framework function so that
asynchronous events can be handled (e.g., speed negotiation, target mode).
Roll internal release tags.
it's already on for the 2XX0) and detect the broken 1040A FIFO. Change
bzero to MEMZERO (portability with **nux). Use memcpy for same reason.
Finally detect QUEUE FULL conditions and return this as an error that
will get cam_periph_error to do it's 'tagged openings now XXX' dance.
suggested compilation cleanups from Eklund. Wire down a hard loop id if we are
not on a platform that has the ability to get to a PCI BIOS (it still will
float to the ID it gets after a LIP but at least we can try). Clarify that the
expanded lun is based upon SCCLUN defines (in f/w).
