Commit Graph

17 Commits

Author SHA1 Message Date
mav
99ff385fd0 Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.

Replace big per-SIM locks with bunch of smaller ones:
 - per-LUN locks to protect device and peripheral drivers state;
 - per-target locks to protect list of LUNs on target;
 - per-bus locks to protect reference counting;
 - per-send queue locks to protect queue of CCBs to be sent;
 - per-done queue locks to protect queue of completed CCBs;
 - remaining per-SIM locks now protect only HBA driver internals.

While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock.  The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded.  Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.

To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.

Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads.  Load balanced
between them using "hash" of the device B:T:L address.

HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.

Sponsored by:	iXsystems, Inc.
MFC after:	2 months
2013-10-21 12:00:26 +00:00
mav
bbfed93309 MFprojects/camlock r249505:
Change CCB queue resize logic to be able safely handle overallocations:
 - (re)allocate queue space in power of 2 chunks with 64 elements minimum
and never shrink it; with only 4/8 bytes per element size is insignificant.
 - automatically reallocate the queue to double size if it is overflowed.
 - if queue reallocation failed, store extra CCBs in unsorted TAILQ,
fetching them back as soon as some queue element is freed.

To free space in CCB for TAILQ linking, change highpowerq from keeping
high-power CCBs to keeping devices frozen due to high-power CCBs.

This encloses all pieces of queue resize logic inside of cam_queue.[ch],
removing some not obvious duties from xpt_release_ccb().
2013-08-05 11:48:40 +00:00
mav
f73c311ca3 MFprojects/camlock r248890, r248897, r248898, r248900, r248903, r248905,
r248917, r248918, r248978, r249001, r249014, r249030:

Remove multilevel freezing mechanism, implemented to handle specifics of
the ATA/SATA error recovery, when post-reset recovery commands should be
allocated when queues are already full of payload requests.  Instead of
removing frozen CCBs with specified range of priorities from the queue
to provide free openings, use simple hack, allowing explicit CCBs over-
allocation for requests with priority higher (numerically lower) then
CAM_PRIORITY_OOB threshold.

Simplify CCB allocation logic by removing SIM-level allocation queue.
After that SIM-level queue manages only CCBs execution, while allocation
logic is localized within each single device.

Suggested by:	gibbs
2013-04-14 09:28:14 +00:00
mav
e867826c02 Increase device CCB queue array size by CAM_RL_VALUES - 1 (4) elements.
It is required to store extra recovery requests in case of bus resets.
On ATA/SATA this fixes assertion panics on HEAD with INVARIANTS enabled or
possible memory corruptions otherwise if timeout/reset happens when device
CCB queue is already full.

Reported by:	gibbs@
MFC after:	1 week
2012-10-11 15:21:07 +00:00
mav
06f4d9dad6 Change queue overflow checks from DIAGNOSTIC+panic() to KASSERT() to make
them enabled on HEAD by default. It is probably better to do single compare
then hunt for unexpected memory corruption.
2012-09-28 12:13:34 +00:00
ed
e97eae1577 Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs.
This means that their use is restricted to a single C file.
2011-11-07 06:44:47 +00:00
mav
3579059b77 MFp4:
Some more missed parts from previous commits.
2009-11-14 20:30:42 +00:00
mav
127ea9fc88 After thinking again, implement cam_ccbq_fini().
This is effectively NULL change, but makes this API a bit more consistent.
2009-10-22 21:07:32 +00:00
avatar
cd4bfad3f4 - Providing fine-grained malloc statistic by replacing M_DEVBUF with
module-specific malloc types.  These should help us to pinpoint the
  possible memory leakage in the future.
- Implementing xpt_alloc_ccb_nowait() and replacing all malloc/free based
  CCB memory management with xpt_alloc_ccb[_nowait]/xpt_free_ccb.  Hopefully
  this would be helpful if someday we move the CCB allocator to use UMA
  instead of malloc().

Encouraged by:	jeffr, rwatson
Reviewed by:	gibbs, scottl
Approved by:	re (scottl)
2005-07-01 15:21:30 +00:00
imp
362fcfc1e2 Start each of the license/copyright comments with /*- 2005-01-05 22:34:37 +00:00
obrien
37ee58d305 Use __FBSDID(). 2003-06-10 17:50:20 +00:00
schweikh
86f7487fb6 Fix typos, mostly s/ an / a / where appropriate and a few s/an/and/
Add FreeBSD Id tag where missing.
2002-12-30 21:18:15 +00:00
peter
3b842d34e8 $Id$ -> $FreeBSD$ 1999-08-28 01:08:13 +00:00
gibbs
47a5eeb679 Fix a long standing bug in the camq teardown code. 1999-08-16 02:23:42 +00:00
gibbs
752ab9a858 cam_periph.c:
Move handling of CAM_AUTOSENSE_FAIL into block dealing with
	all other scsi status errors.

cam_queue.c:
cam_queue.h:
	Fix 'off by one' heap bug in a more efficient manner.  Since
	heap algorithms like to deal with indexes started from 1,
	offset our heap array pointer at allocation time to make this
	so for a C environment.  This makes the implementation of the
	algorithm a bit more efficient.

cam_xpt.c:
	Use macros for accessing the head of the heap so that code
	is isolated from implementation details of the heap.
1999-04-19 21:26:08 +00:00
gibbs
71072fc113 Remove camq_regen(). We already perform modular comparisons
for generation counts, so no further steps to deal with generation
count wrap are required.

Fix an off by one problem in the camq heap code.
1999-04-07 22:57:48 +00:00
gibbs
855593c295 CAM Transport Layer (XPT).
Submitted by:	The CAM Team
1998-09-15 06:33:23 +00:00