Commit Graph

40 Commits

Author SHA1 Message Date
Alexander Motin
321f819ba5 Refactor command ordering/blocking mechanism in CTL.
Replace long per-LUN queue of blocked commands, scanned on each command
completion and sometimes even twice, causing up to O(n^^2) processing cost,
by much shorter per-command blocked queues, scanned only when respective
command completes, and check only commands before the previous blocker,
reducing cost to O(n).

While there, unblock aborted commands to make them "complete" ASAP to be
removed from the OOA queue and so not waste time ordering other commands
against them.  Aborted commands that were not sent to execution yet should
have no visible side effects, so this is safe and easy optimization now,
comparing to commands already in processing, which are a still pain.

Together those two optimizations should fix quite pathological case, when
due to backend slowness CTL accumulated many thousands of blocked requests,
partially aborted by initiator and so supposedly not even existing, but
still wasting CTL CPU time.

MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
2019-02-27 21:29:21 +00:00
Alexander Motin
62e802cf3a Free some space in struct ctl_io_hdr for better use.
- Collapse original_sc and serializing_sc fields into one, since they
are never used simultanously, we have only one local I/O and one remote.

 - Move remote_sglist and local_sglist fields into CTL_PRIV_BACKEND,
since they are used only on Originating SC in XFER mode, where requests
don't ever reach backends, so we can reuse backend's private storage.

MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
2019-02-23 23:35:52 +00:00
Pedro F. Giffuni
bec9534d1d sys/cam: further adoption of SPDX licensing ID tags.
Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.
2017-11-27 15:12:43 +00:00
Alexander Motin
748f120f7c Add sysctl/tunable for maximal request time.
MFC after:	1 week
2017-09-30 13:17:31 +00:00
Edward Tomasz Napierala
5ab1cf33f7 Bring the ctl headers a bit closer to style(9). No functional changes. 2017-01-25 22:52:57 +00:00
Alexander Motin
640603fbf6 Remove writing 'residual' field of struct ctl_scsiio.
This field has no practical use and never readed.  Initiators already
receive respective residual size from frontends.  Removed field had
different semantics, which looks useless, and was never passed through
by any frontend.

While there, fix kern_data_resid field support in case of HA, missed in
r312291.

MFC after:	13 days
2017-01-17 18:32:47 +00:00
Alexander Motin
9cbbfd2f5c Improve use of I/O's private area.
- Since I/Os are allocates from per-port pools, make allocations store
pointer to CTL softc there, and use it where needed instead of global.
 - Created bunch of helper macros to access LUN, port and CTL softc.

MFC after:	 2 weeks
2016-12-29 15:09:34 +00:00
Alexander Motin
e67ac203a6 Allow more efficient use of private area.
There are 16 bytes of space, so we may store two pointers in one.

MFC after:	2 weeks
2016-12-27 23:56:46 +00:00
Alexander Motin
ee5e44e076 Decouple limits on number of LUNs per port and LUs per CTL.
Those two values are not directly related, so make them independent.
This does not change any limits immediately, but makes number of LUNs
per port controllable via tunable/sysctl kern.cam.ctl.lun_map_size.
After this change increasing CTL_MAX_LUNS should be pretty cheap,
and even making it tunable should be easy.

MFC after:	2 weeks
2016-12-27 18:23:16 +00:00
Pedro F. Giffuni
1ffe58516c sys/cam: spelling fixes in comments.
No functional change.
2016-04-29 21:05:48 +00:00
Alexander Motin
648dfc1a29 Umplement media load/eject support for removable devices.
In case of block backend eject really closes the backing store, while
load tries to open it back.  Failed store open is reported as no media.
2015-09-28 20:54:18 +00:00
Alexander Motin
e675024a02 Switch I/O time accounting from system time to uptime.
While there, make num_dmas accounted independently of CTL_TIME_IO.
2015-09-25 10:14:39 +00:00
Alexander Motin
ca85b7c4e9 Synchronize mode pages between HA peers.
We allow to modify only few fields in mode pages now, but still it is
not good if they unexpectedly change during failover.  Also this fixes
reporting of "Mode parameters changed" UAs on secondary node.
2015-09-23 18:33:00 +00:00
Alexander Motin
a85700a912 Make HA peers announce their parameters on connect.
HA protocol requires strict version, parameters and configuration match.
Differences there may cause full set of problems up to kernel panic.
To avoid that, validate peer parameters on connect, and abort connection
immediately if some mismatch detected.
2015-09-23 15:49:27 +00:00
Alexander Motin
b22213694e Remove couple excess SGLIST I/O flags.
Those flags duplicated respective (sg_entries > 0) values.
2015-09-20 10:40:30 +00:00
Alexander Motin
027dd0cfd0 Pack struct ctl_ha_msg_hdr by 8 bytes. 2015-09-19 18:23:06 +00:00
Alexander Motin
75a3108e13 Relax serseq option operation for reads.
Previously, with serseq enabled, next command was unblocked only after
previous completed.  With this change, for read operations, next command
is unblocked as soon as last media read completed.  This is important
for frontends that actually wait for data move completion (like camtgt),
or when data are moved through the HA link, or especially when both.
2015-09-18 19:43:14 +00:00
Alexander Motin
0c05f0dc16 Replicate initiators WWPNs and names between HA peers. 2015-09-17 21:51:11 +00:00
Alexander Motin
54713bce25 Replicate port->init_devid to HA peer. 2015-09-17 20:36:34 +00:00
Alexander Motin
6213882769 When reporting TPT UA, report which of thresholds was reached. 2015-09-17 17:00:36 +00:00
Alexander Motin
7b6660c9aa Remove CTL_PRIV_LBA_LEN from HA messages.
Previously it was used for statistics, but now just a 16 extra bytes.
2015-09-14 09:56:01 +00:00
Alexander Motin
ceff31dc0c Implement QUERY TASK, QUERY TASK SET and QUERY ASYNC EVENT.
Now we support most of SAM-5 task management.
2015-09-14 08:01:05 +00:00
Alexander Motin
7ac58230ea Reimplement CTL High Availability.
CTL HA functionality was originally implemented by Copan many years ago,
but large part of the sources was never published.  This change includes
clean room implementation of the missing code and fixes for many bugs.

This code supports dual-node HA with ALUA in four modes:
 - Active/Unavailable without interlink between nodes;
 - Active/Standby with second node handling only basic LUN discovery and
reservation, synchronizing with the first node through the interlink;
 - Active/Active with both nodes processing commands and accessing the
backing storage, synchronizing with the first node through the interlink;
 - Active/Active with second node working as proxy, transfering all
commands to the first node for execution through the interlink.

Unlike original Copan's implementation, depending on specific hardware,
this code uses simple custom TCP-based protocol for interlink.  It has
no authentication, so it should never be enabled on public interfaces.

The code may still need some polishing, but generally it is functional.

Relnotes:	yes
Sponsored by:	iXsystems, Inc.
2015-09-10 12:40:31 +00:00
Alexander Motin
fb606ebabc Remove unused target and initiator IDs. 2015-09-10 10:46:21 +00:00
Alexander Motin
3f326305fc Disable CTL_IO_DELAY feature.
It is too developer-oriented to be enabled by default.
2015-09-10 09:27:22 +00:00
Alexander Motin
7278725be5 Remove 600 bytes of port_priv from struct ctl_io_hdr.
This field used only for camtgt frontend, and once it any way preallocates
all requests, let it preallocate this memory too, not bothering core code.
2015-08-29 15:33:31 +00:00
Alexander Motin
f7241cceb0 Coalesce last data move and command status for read commands.
Make CTL core and block backend set success status before initiating last
data move for read commands.  Make CAM target and iSCSI frontends detect
such condition and send command status together with data.  New I/O flag
allows to skip duplicate status sending on later fe_done() call.

For Fibre Channel this change saves one of three interrupts per read command,
increasing performance from 126K to 160K IOPS.  For iSCSI this change saves
one of three PDUs per read command, increasing performance from 1M to 1.2M
IOPS.

MFC after:	1 month
Sponsored by:	iXsystems, Inc.
2014-11-25 17:53:35 +00:00
Alexander Motin
1251a76b12 Replace home-grown CTL IO allocator with UMA.
Old allocator created significant lock congestion protecting its lists
of preallocated I/Os, while UMA provides much better SMP scalability.
The downside of UMA is lack of reliable preallocation, that could guarantee
successful allocation in non-sleepable environments.  But careful code
review shown, that only CAM target frontend really has that requirement.
Fix that making that frontend preallocate and statically bind CTL I/O for
every ATIO/INOT it preallocates any way.  That allows to avoid allocations
in hot I/O path.  Other frontends either may sleep in allocation context
or can properly handle allocation errors.

On 40-core server with 6 ZVOL-backed LUNs and 7 iSCSI client connections
this change increases peak performance from ~700K to >1M IOPS!  Yay! :)

MFC after:	1 month
Sponsored by:	iXsystems, Inc.
2014-11-24 11:37:27 +00:00
Alexander Motin
9a0190c9a1 Remove couple Copan's vendor-specific mode pages.
Those pages are highly system-/hardware-specific, the code is incomplete,
and so they hardly can be useful for anybody else.
2014-10-14 11:28:25 +00:00
Alexander Motin
55551d0542 Improve cache control support, including DPO/FUA flags and the mode page.
At this moment it works only for files and ZVOLs in device mode since BIOs
have no respective respective cache control flags (DPO/FUA).

MFC after:	1 month
Sponsored by:	iXsystems, Inc.
2014-09-09 11:38:29 +00:00
Alexander Motin
8cbf9eae6f Increase maximal number of SCSI ports in CTL from 32 to 128.
After I gave each iSCSI target its own port, the old limit appeared to be
not so big.  This change almost proportionally increases per-LUN memory
use, but it is still three times better then it was before r268807.

MFC after:	2 weeks
2014-07-17 21:16:52 +00:00
Alexander Motin
b33b96e352 Enable TAS feature: notify initiator if its command was aborted by other.
That should make operation more kind to multi-initiator environment.
Without this, other initiators may find out that something bad happened
to their commands only via command timeout.
2014-07-08 16:38:05 +00:00
Alexander Motin
0020682baa Implement ABORT TASK SET and I_T NEXUS RESET task management functions.
Use the last one to terminate active commands on iSCSI session termination.
Previous code was aborting only commands doing some data moves.
2014-07-07 03:10:56 +00:00
Alexander Motin
c26eee2dc9 Move lun_map() method from command nexus to port.
Previous implementation made impossible to do some things, such as calling
it for ports other then one through which command arrived.
2014-07-06 06:21:34 +00:00
Alexander Motin
3a8ce4a36b Introduce fine-grained CTL locking to improve SMP scalability.
Split global ctl_lock, historically protecting most of CTL context:
 - remaining ctl_lock now protects lists of fronends and backends;
 - per-LUN lun_lock(s) protect LUN-specific information;
 - per-thread queue_lock(s) protect request queues.
This allows to radically reduce congestion on ctl_lock.

Create multiple worker threads, depending on number of CPUs, and assign
each LUN to one of them.  This allows to spread load between multiple CPUs,
still avoiging congestion on queues and LUNs locks.

On 40-core server, exporting 5 LUNs, each backed by gstripe of SATA SSDs,
accessed via 6 iSCSI connections, this change improves peak request rate
from 250K to 680K IOPS.

MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
2014-06-25 17:02:01 +00:00
Alexander Motin
11b569f7cb Add support for VERIFY(10/12/16) and COMPARE AND WRITE SCSI commands.
Make data_submit backends method support not only read and write requests,
but also two new ones: verify and compare.  Verify just checks readability
of the data in specified location without transferring them outside.
Compare reads the specified data and compares them to received data,
returning error if they are different.

VERIFY(10/12/16) commands request either verify or compare from backend,
depending on BYTCHK CDB field.  COMPARE AND WRITE command executed in two
stages: first it requests compare, and then, if succeesed, requests write.
Atomicity of operation is guarantied by CTL request ordering code.

MFC after:	2 weeks
Sponsored by:	iXsystems, Inc.
2014-06-16 11:00:14 +00:00
Edward Tomasz Napierala
8cd22f5edf Provide better descriptions for 'struct ctl_scsiio' fields; based mostly
on emails from ken@.
2014-05-04 15:35:04 +00:00
Alexander Motin
ee7f31c068 Add support for SCSI UNMAP commands to CTL.
This patch adds support for three new SCSI commands: UNMAP, WRITE SAME(10)
and WRITE SAME(16).  WRITE SAME commands support both normal write mode
and UNMAP flag.  To properly report UNMAP capabilities this patch also adds
support for reporting two new VPD pages: Block limits and Logical Block
Provisioning.

UNMAP support can be enabled per-LUN by adding "-o unmap=on" to `ctladm
create` command line or "option unmap on" to lun sections of /etc/ctl.conf.

At this moment UNMAP supported for ramdisks and device-backed block LUNs.
It was tested to work great with ZFS ZVOLs.  For file-backed LUNs UNMAP
support is unfortunately missing due to absence of respective VFS KPI.

Reviewed by:	ken
MFC after:	1 month
Sponsored by:	iXsystems, Inc
2014-04-08 20:50:48 +00:00
Edward Tomasz Napierala
81a2151d5c CTL changes required for iSCSI target, most notably LUN remapping
and a mechanism to allow CTL frontends for retrieving LUN options.

Reviewed by:	ken (earlier version)
2013-08-24 01:50:31 +00:00
Kenneth D. Merry
130f4520cb Add the CAM Target Layer (CTL).
CTL is a disk and processor device emulation subsystem originally written
for Copan Systems under Linux starting in 2003.  It has been shipping in
Copan (now SGI) products since 2005.

It was ported to FreeBSD in 2008, and thanks to an agreement between SGI
(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is
available under a BSD-style license.  The intent behind the agreement was
that Spectra would work to get CTL into the FreeBSD tree.

Some CTL features:

 - Disk and processor device emulation.
 - Tagged queueing
 - SCSI task attribute support (ordered, head of queue, simple tags)
 - SCSI implicit command ordering support.  (e.g. if a read follows a mode
   select, the read will be blocked until the mode select completes.)
 - Full task management support (abort, LUN reset, target reset, etc.)
 - Support for multiple ports
 - Support for multiple simultaneous initiators
 - Support for multiple simultaneous backing stores
 - Persistent reservation support
 - Mode sense/select support
 - Error injection support
 - High Availability support (1)
 - All I/O handled in-kernel, no userland context switch overhead.

(1) HA Support is just an API stub, and needs much more to be fully
    functional.

ctl.c:			The core of CTL.  Command handlers and processing,
			character driver, and HA support are here.

ctl.h:			Basic function declarations and data structures.

ctl_backend.c,
ctl_backend.h:		The basic CTL backend API.

ctl_backend_block.c,
ctl_backend_block.h:	The block and file backend.  This allows for using
			a disk or a file as the backing store for a LUN.
			Multiple threads are started to do I/O to the
			backing device, primarily because the VFS API
			requires that to get any concurrency.

ctl_backend_ramdisk.c:	A "fake" ramdisk backend.  It only allocates a
			small amount of memory to act as a source and sink
			for reads and writes from an initiator.  Therefore
			it cannot be used for any real data, but it can be
			used to test for throughput.  It can also be used
			to test initiators' support for extremely large LUNs.

ctl_cmd_table.c:	This is a table with all 256 possible SCSI opcodes,
			and command handler functions defined for supported
			opcodes.

ctl_debug.h:		Debugging support.

ctl_error.c,
ctl_error.h:		CTL-specific wrappers around the CAM sense building
			functions.

ctl_frontend.c,
ctl_frontend.h:		These files define the basic CTL frontend port API.

ctl_frontend_cam_sim.c:	This is a CTL frontend port that is also a CAM SIM.
			This frontend allows for using CTL without any
			target-capable hardware.  So any LUNs you create in
			CTL are visible in CAM via this port.

ctl_frontend_internal.c,
ctl_frontend_internal.h:
			This is a frontend port written for Copan to do
			some system-specific tasks that required sending
			commands into CTL from inside the kernel.  This
			isn't entirely relevant to FreeBSD in general,
			but can perhaps be repurposed.

ctl_ha.h:		This is a stubbed-out High Availability API.  Much
			more is needed for full HA support.  See the
			comments in the header and the description of what
			is needed in the README.ctl.txt file for more
			details.

ctl_io.h:		This defines most of the core CTL I/O structures.
			union ctl_io is conceptually very similar to CAM's
			union ccb.

ctl_ioctl.h:		This defines all ioctls available through the CTL
			character device, and the data structures needed
			for those ioctls.

ctl_mem_pool.c,
ctl_mem_pool.h:		Generic memory pool implementation used by the
			internal frontend.

ctl_private.h:		Private data structres (e.g. CTL softc) and
			function prototypes.  This also includes the SCSI
			vendor and product names used by CTL.

ctl_scsi_all.c,
ctl_scsi_all.h:		CTL wrappers around CAM sense printing functions.

ctl_ser_table.c:	Command serialization table.  This defines what
			happens when one type of command is followed by
			another type of command.

ctl_util.c,
ctl_util.h:		CTL utility functions, primarily designed to be
			used from userland.  See ctladm for the primary
			consumer of these functions.  These include CDB
			building functions.

scsi_ctl.c:		CAM target peripheral driver and CTL frontend port.
			This is the path into CTL for commands from
			target-capable hardware/SIMs.

README.ctl.txt:		CTL code features, roadmap, to-do list.

usr.sbin/Makefile:	Add ctladm.

ctladm/Makefile,
ctladm/ctladm.8,
ctladm/ctladm.c,
ctladm/ctladm.h,
ctladm/util.c:		ctladm(8) is the CTL management utility.
			It fills a role similar to camcontrol(8).
			It allow configuring LUNs, issuing commands,
			injecting errors and various other control
			functions.

usr.bin/Makefile:	Add ctlstat.

ctlstat/Makefile
ctlstat/ctlstat.8,
ctlstat/ctlstat.c:	ctlstat(8) fills a role similar to iostat(8).
			It reports I/O statistics for CTL.

sys/conf/files:		Add CTL files.

sys/conf/NOTES:		Add device ctl.

sys/cam/scsi_all.h:	To conform to more recent specs, the inquiry CDB
			length field is now 2 bytes long.

			Add several mode page definitions for CTL.

sys/cam/scsi_all.c:	Handle the new 2 byte inquiry length.

sys/dev/ciss/ciss.c,
sys/dev/ata/atapi-cam.c,
sys/cam/scsi/scsi_targ_bh.c,
scsi_target/scsi_cmds.c,
mlxcontrol/interface.c:	Update for 2 byte inquiry length field.

scsi_da.h:		Add versions of the format and rigid disk pages
			that are in a more reasonable format for CTL.

amd64/conf/GENERIC,
i386/conf/GENERIC,
ia64/conf/GENERIC,
sparc64/conf/GENERIC:	Add device ctl.

i386/conf/PAE:		The CTL frontend SIM at least does not compile
			cleanly on PAE.

Sponsored by:	Copan Systems, SGI and Spectra Logic
MFC after:	1 month
2012-01-12 00:34:33 +00:00