439 lines
15 KiB
Plaintext
439 lines
15 KiB
Plaintext
/* $FreeBSD$ */
|
|
|
|
CTL - CAM Target Layer Description
|
|
|
|
Revision 1.4 (December 29th, 2011)
|
|
Ken Merry <ken@FreeBSD.org>
|
|
|
|
Table of Contents:
|
|
=================
|
|
|
|
Introduction
|
|
Features
|
|
Configuring and Running CTL
|
|
Revision 1.N Changes
|
|
To Do List
|
|
Code Roadmap
|
|
Userland Commands
|
|
|
|
Introduction:
|
|
============
|
|
|
|
CTL is a disk and processor device emulation subsystem originally written
|
|
for Copan Systems under Linux starting in 2003. It has been shipping in
|
|
Copan (now SGI) products since 2005.
|
|
|
|
It was ported to FreeBSD in 2008, and thanks to an agreement between SGI
|
|
(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is
|
|
available under a BSD-style license. The intent behind the agreement was
|
|
that Spectra would work to get CTL into the FreeBSD tree.
|
|
|
|
Features:
|
|
========
|
|
|
|
- Disk and processor device emulation.
|
|
- Tagged queueing
|
|
- SCSI task attribute support (ordered, head of queue, simple tags)
|
|
- SCSI implicit command ordering support. (e.g. if a read follows a mode
|
|
select, the read will be blocked until the mode select completes.)
|
|
- Full task management support (abort, LUN reset, target reset, etc.)
|
|
- Support for multiple ports
|
|
- Support for multiple simultaneous initiators
|
|
- Support for multiple simultaneous backing stores
|
|
- Persistent reservation support
|
|
- Mode sense/select support
|
|
- Error injection support
|
|
- High Availability support (1)
|
|
- All I/O handled in-kernel, no userland context switch overhead.
|
|
|
|
(1) HA Support is just an API stub, and needs much more to be fully
|
|
functional. See the to-do list below.
|
|
|
|
Configuring and Running CTL:
|
|
===========================
|
|
|
|
- After applying the CTL patchset to your tree, build world and install it
|
|
on your target system.
|
|
|
|
- Add 'device ctl' to your kernel configuration file.
|
|
|
|
- If you're running with a 8Gb or 4Gb Qlogic FC board, add
|
|
'options ISP_TARGET_MODE' to your kernel config file. Keep in mind that
|
|
the isp(4) driver can run in target or initiator mode, but not both on
|
|
the same machine. 'device ispfw' or loading the ispfw module is also
|
|
recommended.
|
|
|
|
- Rebuild and install a new kernel.
|
|
|
|
- Reboot with the new kernel.
|
|
|
|
- To add a LUN with the RAM disk backend:
|
|
|
|
ctladm create -b ramdisk -s 10485760000000000000
|
|
ctladm port -o on
|
|
|
|
- You should now see the CTL disk LUN through camcontrol devlist:
|
|
|
|
scbus6 on ctl2cam0 bus 0:
|
|
<FREEBSD CTLDISK 0001> at scbus6 target 1 lun 0 (da24,pass32)
|
|
<> at scbus6 target -1 lun -1 ()
|
|
|
|
This is visible through the CTL CAM SIM. This allows using CTL without
|
|
any physical hardware. You should be able to issue any normal SCSI
|
|
commands to the device via the pass(4)/da(4) devices.
|
|
|
|
If any target-capable HBAs are in the system (e.g. isp(4)), and have
|
|
target mode enabled, you should now also be able to see the CTL LUNs via
|
|
that target interface.
|
|
|
|
Note that all CTL LUNs are presented to all frontends. There is no
|
|
LUN masking, or separate, per-port configuration.
|
|
|
|
- Note that the ramdisk backend is a "fake" ramdisk. That is, it is
|
|
backed by a small amount of RAM that is used for all I/O requests. This
|
|
is useful for performance testing, but not for any data integrity tests.
|
|
|
|
- To add a LUN with the block/file backend:
|
|
|
|
truncate -s +1T myfile
|
|
ctladm create -b block -o file=myfile
|
|
ctladm port -o on
|
|
|
|
- You can also see a list of LUNs and their backends like this:
|
|
|
|
# ctladm devlist
|
|
LUN Backend Size (Blocks) BS Serial Number Device ID
|
|
0 block 2147483648 512 MYSERIAL 0 MYDEVID 0
|
|
1 block 2147483648 512 MYSERIAL 1 MYDEVID 1
|
|
2 block 2147483648 512 MYSERIAL 2 MYDEVID 2
|
|
3 block 2147483648 512 MYSERIAL 3 MYDEVID 3
|
|
4 block 2147483648 512 MYSERIAL 4 MYDEVID 4
|
|
5 block 2147483648 512 MYSERIAL 5 MYDEVID 5
|
|
6 block 2147483648 512 MYSERIAL 6 MYDEVID 6
|
|
7 block 2147483648 512 MYSERIAL 7 MYDEVID 7
|
|
8 block 2147483648 512 MYSERIAL 8 MYDEVID 8
|
|
9 block 2147483648 512 MYSERIAL 9 MYDEVID 9
|
|
10 block 2147483648 512 MYSERIAL 10 MYDEVID 10
|
|
11 block 2147483648 512 MYSERIAL 11 MYDEVID 11
|
|
|
|
- You can see the LUN type and backing store for block/file backend LUNs
|
|
like this:
|
|
|
|
# ctladm devlist -v
|
|
LUN Backend Size (Blocks) BS Serial Number Device ID
|
|
0 block 2147483648 512 MYSERIAL 0 MYDEVID 0
|
|
lun_type=0
|
|
num_threads=14
|
|
file=testdisk0
|
|
1 block 2147483648 512 MYSERIAL 1 MYDEVID 1
|
|
lun_type=0
|
|
num_threads=14
|
|
file=testdisk1
|
|
2 block 2147483648 512 MYSERIAL 2 MYDEVID 2
|
|
lun_type=0
|
|
num_threads=14
|
|
file=testdisk2
|
|
3 block 2147483648 512 MYSERIAL 3 MYDEVID 3
|
|
lun_type=0
|
|
num_threads=14
|
|
file=testdisk3
|
|
4 block 2147483648 512 MYSERIAL 4 MYDEVID 4
|
|
lun_type=0
|
|
num_threads=14
|
|
file=testdisk4
|
|
5 block 2147483648 512 MYSERIAL 5 MYDEVID 5
|
|
lun_type=0
|
|
num_threads=14
|
|
file=testdisk5
|
|
6 block 2147483648 512 MYSERIAL 6 MYDEVID 6
|
|
lun_type=0
|
|
num_threads=14
|
|
file=testdisk6
|
|
7 block 2147483648 512 MYSERIAL 7 MYDEVID 7
|
|
lun_type=0
|
|
num_threads=14
|
|
file=testdisk7
|
|
8 block 2147483648 512 MYSERIAL 8 MYDEVID 8
|
|
lun_type=0
|
|
num_threads=14
|
|
file=testdisk8
|
|
9 block 2147483648 512 MYSERIAL 9 MYDEVID 9
|
|
lun_type=0
|
|
num_threads=14
|
|
file=testdisk9
|
|
10 ramdisk 0 0 MYSERIAL 0 MYDEVID 0
|
|
lun_type=3
|
|
11 ramdisk 204800000000000 512 MYSERIAL 1 MYDEVID 1
|
|
lun_type=0
|
|
|
|
|
|
Revision 1.4 Changes
|
|
====================
|
|
- Added in the second HA mode (where CTL does the data transfers instead
|
|
of having data transfers done below CTL), and abstracted out the Copan
|
|
HA API.
|
|
|
|
- Fixed the phantom device problem in the CTL CAM SIM and improved the
|
|
CAM SIM to automatically trigger a rescan when the port is enabled and
|
|
disabled.
|
|
|
|
- Made the number of threads in the block backend configurable via sysctl,
|
|
loader tunable and the ctladm command line. (You can now specify
|
|
-o num_threads=4 when creating a LUN with ctladm create.)
|
|
|
|
- Fixed some LUN selection issues in ctlstat(8) and allowed for selection
|
|
of LUN numbers up to 1023.
|
|
|
|
- General cleanup.
|
|
|
|
- This version intended for public release.
|
|
|
|
Revision 1.3 Changes
|
|
====================
|
|
- Added descriptor sense support to CTL. It can be enabled through the
|
|
control mode page (10), but is disabled by default.
|
|
|
|
- Improved error injection support. The number of errors that can be
|
|
injected with 'ctladm inject' has been increased, and any arbitrary
|
|
sense data may now be injected as well.
|
|
|
|
- The port infrastructure has been revamped. Individual ports and types
|
|
of ports may now be enabled and disabled from the command line. ctladm
|
|
now has the ability to set the WWNN and WWPN for each port.
|
|
|
|
- The block backend can now send multiple I/Os to backing files. Multiple
|
|
writes are only allowed for ZFS, but multiple readers are allowed for
|
|
any filesystem.
|
|
|
|
- The block and ramdisk backends now support setting the LUN blocksize.
|
|
There are some restrictions when the backing device is a block device,
|
|
but otherwise the blocksize may be set to anything.
|
|
|
|
Revision 1.2 Changes
|
|
====================
|
|
|
|
- CTL initialization process has been revamped. Instead of using an
|
|
ad-hoc method, it is now sequenced through SYSINIT() calls.
|
|
|
|
- A block/file backend has been added. This allows using arbitrary files
|
|
or block devices as a backing store.
|
|
|
|
- The userland LUN configuration interface has been completely rewritten.
|
|
Configuration is now done out of band.
|
|
|
|
- The ctladm(8) command line interface has been revamped, and is now
|
|
similar to camcontrol(8).
|
|
|
|
To Do List:
|
|
==========
|
|
|
|
- Use devstat(9) for CTL's statistics collection. CTL uses a home-grown
|
|
statistics collection system that is similar to devstat(9). ctlstat
|
|
should be retired in favor of iostat, etc., once aggregation modes are
|
|
available in iostat to match the behavior of ctlstat -t and dump modes
|
|
are available to match the behavior of ctlstat -d/ctlstat -J.
|
|
|
|
- ZFS ARC backend for CTL. Since ZFS copies all I/O into the ARC
|
|
(Adaptive Replacement Cache), running the block/file backend on top of a
|
|
ZFS-backed zdev or file will involve an extra set of copies. The
|
|
optimal solution for backing targets served by CTL with ZFS would be to
|
|
allocate buffers out of the ARC directly, and DMA to/from them directly.
|
|
That would eliminate an extra data buffer allocation and copy.
|
|
|
|
- Switch CTL over to using CAM CCBs instead of its own union ctl_io. This
|
|
will likely require a significant amount of work, but will eliminate
|
|
another data structure in the stack, more memory allocations, etc. This
|
|
will also require changes to the CAM CCB structure to support CTL.
|
|
|
|
- Full-featured High Availability support. The HA API that is in ctl_ha.h
|
|
is essentially a renamed version of Copan's HA API. There is no
|
|
substance to it, but it remains in CTL to show what needs to be done to
|
|
implement active/active HA from a CTL standpoint. The things that would
|
|
need to be done include:
|
|
- A kernel level software API for message passing as well as DMA
|
|
between at least two nodes.
|
|
- Hardware support and drivers for inter-node communication. This
|
|
could be as simples as ethernet hardware and drivers.
|
|
- A "supervisor", or startup framework to control and coordinate
|
|
HA startup, failover (going from active/active to single mode),
|
|
and failback (going from single mode to active/active).
|
|
- HA support in other components of the stack. The goal behind HA
|
|
is that one node can fail and another node can seamlessly take
|
|
over handling I/O requests. This requires support from pretty
|
|
much every component in the storage stack, from top to bottom.
|
|
CTL is one piece of it, but you also need support in the RAID
|
|
stack/filesystem/backing store. You also need full configuration
|
|
mirroring, and all peer nodes need to be able to talk to the
|
|
underlying storage hardware.
|
|
|
|
Code Roadmap:
|
|
============
|
|
|
|
CTL has the concept of pluggable frontend ports and backends. All
|
|
frontends and backends can be active at the same time. You can have a
|
|
ramdisk-backed LUN present along side a file backed LUN.
|
|
|
|
ctl.c:
|
|
-----
|
|
|
|
This is the core of CTL, where all of the command handlers and a lot of
|
|
other things live. Yes, it is large. It started off small and grew to its
|
|
current size over time. Perhaps it can be split into more files at some
|
|
point.
|
|
|
|
Here is a roadmap of some of the primary functions in ctl.c. Starting here
|
|
and following the various leaf functions will show the command flow.
|
|
|
|
ctl_queue() This is where commands from the frontend ports come
|
|
in.
|
|
|
|
ctl_queue_sense() This is only used for non-packetized SCSI. i.e.
|
|
parallel SCSI prior to U320 and perhaps U160.
|
|
|
|
ctl_work_thread() This is the primary work thread, and everything gets
|
|
executed from there.
|
|
|
|
ctl_scsiio_precheck() This where all of the initial checks are done, and I/O
|
|
is either queued for execution or blocked.
|
|
|
|
ctl_scsiio() This is where the command handler is actually
|
|
executed. (See ctl_cmd_table.c for the mapping of
|
|
SCSI opcode to command handler function.)
|
|
|
|
ctl_done() This is the routine called (or ctl_done_lock()) to
|
|
initiate the command completion process.
|
|
|
|
ctl_process_done() This is where command completion actually happens.
|
|
|
|
ctl.h:
|
|
-----
|
|
|
|
Basic function declarations and data structures.
|
|
|
|
ctl_backend.c,
|
|
ctl_backend.h:
|
|
-------------
|
|
|
|
These files define the basic CTL backend API. The comments in the header
|
|
explain the API.
|
|
|
|
ctl_backend_block.c
|
|
ctl_backend_block.h:
|
|
-------------------
|
|
|
|
The block and file backend. This allows for using a disk or a file as the
|
|
backing store for a LUN. Multiple threads are started to do I/O to the
|
|
backing device, primarily because the VFS API requires that to get any
|
|
concurrency.
|
|
|
|
ctl_backend_ramdisk.c:
|
|
---------------------
|
|
|
|
A "fake" ramdisk backend. It only allocates a small amount of memory to
|
|
act as a source and sink for reads and writes from an initiator. Therefore
|
|
it cannot be used for any real data, but it can be used to test for
|
|
throughput. It can also be used to test initiators' support for extremely
|
|
large LUNs.
|
|
|
|
ctl_cmd_table.c:
|
|
---------------
|
|
|
|
This is a table with all 256 possible SCSI opcodes, and command handler
|
|
functions defined for supported opcodes. It is included in ctl.c.
|
|
|
|
ctl_debug.h:
|
|
-----------
|
|
|
|
Simplistic debugging support.
|
|
|
|
ctl_error.c,
|
|
ctl_error.h:
|
|
-----------
|
|
|
|
CTL-specific wrappers around the CAM sense building functions.
|
|
|
|
ctl_frontend.c,
|
|
ctl_frontend.h:
|
|
--------------
|
|
|
|
These files define the basic CTL frontend port API. The comments in the
|
|
header explain the API.
|
|
|
|
ctl_frontend_cam_sim.c:
|
|
----------------------
|
|
|
|
This is a CTL frontend port that is also a CAM SIM. The idea is that this
|
|
frontend allows for using CTL without any target-capable hardware. So any
|
|
LUNs you create in CTL are visible via this port.
|
|
|
|
|
|
ctl_frontend_internal.c
|
|
ctl_frontend_internal.h:
|
|
-----------------------
|
|
|
|
This is a frontend port written for Copan to do some system-specific tasks
|
|
that required sending commands into CTL from inside the kernel. This isn't
|
|
entirely relevant to FreeBSD in general, but can perhaps be repurposed or
|
|
removed later.
|
|
|
|
ctl_ha.h:
|
|
--------
|
|
|
|
This is a stubbed-out High Availability API. See the comments in the
|
|
header and the description of what is needed as far as HA support above.
|
|
|
|
ctl_io.h:
|
|
--------
|
|
|
|
This defines most of the core CTL I/O structures. union ctl_io is
|
|
conceptually very similar to CAM's union ccb.
|
|
|
|
ctl_ioctl.h:
|
|
-----------
|
|
|
|
This defines all ioctls available through the CTL character device, and
|
|
the data structures needed for those ioctls.
|
|
|
|
ctl_private.h:
|
|
-------------
|
|
|
|
Private data structres (e.g. CTL softc) and function prototypes. This also
|
|
includes the SCSI vendor and product names used by CTL.
|
|
|
|
ctl_scsi_all.c
|
|
ctl_scsi_all.h:
|
|
--------------
|
|
|
|
CTL wrappers around CAM sense printing functions.
|
|
|
|
ctl_ser_table.c:
|
|
---------------
|
|
|
|
Command serialization table. This defines what happens when one type of
|
|
command is followed by another type of command. e.g., what do you do when
|
|
you have a mode select followed by a write? You block the write until the
|
|
mode select is complete. That is defined in this table.
|
|
|
|
ctl_util.c
|
|
ctl_util.h:
|
|
----------
|
|
|
|
CTL utility functions, primarily designed to be used from userland. See
|
|
ctladm for the primary consumer of these functions. These include CDB
|
|
building functions.
|
|
|
|
scsi_ctl.c:
|
|
----------
|
|
|
|
CAM target peripheral driver and CTL frontend port. This is the path into
|
|
CTL for commands from target-capable hardware/SIMs.
|
|
|
|
Userland Commands:
|
|
=================
|
|
|
|
ctladm(8) fills a role similar to camcontrol(8). It allow configuring LUNs,
|
|
issuing commands, injecting errors and various other control functions.
|
|
|
|
ctlstat(8) fills a role similar to iostat(8). It reports I/O statistics
|
|
for CTL.
|