d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

MFC r287621: Reimplement CTL High Availability.

Browse Source 
CTL HA functionality was originally implemented by Copan many years ago,
but large part of the sources was never published.  This change includes
clean room implementation of the missing code and fixes for many bugs.

This code supports dual-node HA with ALUA in four modes:
 - Active/Unavailable without interlink between nodes;
 - Active/Standby with second node handling only basic LUN discovery and
reservation, synchronizing with the first node through the interlink;
 - Active/Active with both nodes processing commands and accessing the
backing storage, synchronizing with the first node through the interlink;
 - Active/Active with second node working as proxy, transfering all
commands to the first node for execution through the interlink.

Unlike original Copan's implementation, depending on specific hardware,
this code uses simple custom TCP-based protocol for interlink.  It has
no authentication, so it should never be enabled on public interfaces.

The code may still need some polishing, but generally it is functional.

Relnotes:       yes
Sponsored by:   iXsystems, Inc.
This commit is contained in:
mav 2015-10-05 08:57:16 +00:00
parent 63b0da2b15
commit 568256e886
22 changed files with 2401 additions and 1994 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
sys/cam/ctl/README.ctl.txt
View File

@ -43,12 +43,9 @@ Features:
- Persistent reservation support
- Mode sense/select support
- Error injection support
- High Availability support (1)
- High Availability support
- All I/O handled in-kernel, no userland context switch overhead.
(1) HA Support is just an API stub, and needs much more to be fully
functional. See the to-do list below.
Configuring and Running CTL:
===========================
@ -245,27 +242,6 @@ To Do List:
another data structure in the stack, more memory allocations, etc. This
will also require changes to the CAM CCB structure to support CTL.
- Full-featured High Availability support. The HA API that is in ctl_ha.h
is essentially a renamed version of Copan's HA API. There is no
substance to it, but it remains in CTL to show what needs to be done to
implement active/active HA from a CTL standpoint. The things that would
need to be done include:
- A kernel level software API for message passing as well as DMA
between at least two nodes.
- Hardware support and drivers for inter-node communication. This
could be as simples as ethernet hardware and drivers.
- A "supervisor", or startup framework to control and coordinate
HA startup, failover (going from active/active to single mode),
and failback (going from single mode to active/active).
- HA support in other components of the stack. The goal behind HA
is that one node can fail and another node can seamlessly take
over handling I/O requests. This requires support from pretty
much every component in the storage stack, from top to bottom.
CTL is one piece of it, but you also need support in the RAID
stack/filesystem/backing store. You also need full configuration
mirroring, and all peer nodes need to be able to talk to the
underlying storage hardware.
Code Roadmap:
============
@ -365,11 +341,11 @@ This is a CTL frontend port that is also a CAM SIM. The idea is that this
frontend allows for using CTL without any target-capable hardware. So any
LUNs you create in CTL are visible via this port.
ctl_ha.c:
ctl_ha.h:
--------
This is a stubbed-out High Availability API. See the comments in the
header and the description of what is needed as far as HA support above.
This is a High Availability API and TCP-based interlink implementation.
ctl_io.h:
--------

2739
sys/cam/ctl/ctl.c
View File

File diff suppressed because it is too large Load Diff

26
sys/cam/ctl/ctl.h
View File

@ -138,25 +138,13 @@ struct ctl_page_index;
SYSCTL_DECL(_kern_cam_ctl);
#endif
/*
* Call these routines to enable or disable front end ports.
*/
int ctl_port_enable(ctl_port_type port_type);
int ctl_port_disable(ctl_port_type port_type);
/*
* This routine grabs a list of frontend ports.
*/
int ctl_port_list(struct ctl_port_entry *entries, int num_entries_alloced,
int *num_entries_filled, int *num_entries_dropped,
ctl_port_type port_type, int no_virtual);
/*
* Put a string into an sbuf, escaping characters that are illegal or not
* recommended in XML. Note this doesn't escape everything, just > < and &.
*/
int ctl_sbuf_printf_esc(struct sbuf *sb, char *str, int size);
int ctl_ffz(uint32_t *mask, uint32_t size);
int ctl_ffz(uint32_t *mask, uint32_t first, uint32_t last);
int ctl_set_mask(uint32_t *mask, uint32_t bit);
int ctl_clear_mask(uint32_t *mask, uint32_t bit);
int ctl_is_set(uint32_t *mask, uint32_t bit);
@ -165,11 +153,6 @@ int ctl_caching_sp_handler(struct ctl_scsiio *ctsio,
int ctl_control_page_handler(struct ctl_scsiio *ctsio,
struct ctl_page_index *page_index,
uint8_t *page_ptr);
/**
int ctl_failover_sp_handler(struct ctl_scsiio *ctsio,
struct ctl_page_index *page_index,
uint8_t *page_ptr);
**/
int ctl_debugconf_sp_sense_handler(struct ctl_scsiio *ctsio,
struct ctl_page_index *page_index,
int pc);
@ -189,11 +172,12 @@ void ctl_data_submit_done(union ctl_io *io);
void ctl_config_read_done(union ctl_io *io);
void ctl_config_write_done(union ctl_io *io);
void ctl_portDB_changed(int portnum);
#ifdef notyet
void ctl_init_isc_msg(void);
#endif
int ctl_ioctl_io(struct cdev *dev, u_long cmd, caddr_t addr, int flag,
struct thread *td);
struct ctl_lun;
void ctl_isc_announce_lun(struct ctl_lun *lun);
struct ctl_port;
void ctl_isc_announce_port(struct ctl_port *port);
/*
* KPI to manipulate LUN/port options

6
sys/cam/ctl/ctl_backend.h
View File

@ -307,6 +307,12 @@ int ctl_lun_operable(struct ctl_be_lun *be_lun);
int ctl_lun_offline(struct ctl_be_lun *be_lun);
int ctl_lun_online(struct ctl_be_lun *be_lun);
/*
* Called on LUN HA role change.
*/
int ctl_lun_primary(struct ctl_be_lun *be_lun);
int ctl_lun_secondary(struct ctl_be_lun *be_lun);
/*
* Let the backend notify the initiator about changed capacity.
*/

80
sys/cam/ctl/ctl_backend_block.c
View File

@ -87,7 +87,9 @@ __FBSDID("$FreeBSD$");
#include <cam/ctl/ctl.h>
#include <cam/ctl/ctl_backend.h>
#include <cam/ctl/ctl_ioctl.h>
#include <cam/ctl/ctl_ha.h>
#include <cam/ctl/ctl_scsi_all.h>
#include <cam/ctl/ctl_private.h>
#include <cam/ctl/ctl_error.h>
/*
@ -219,6 +221,8 @@ struct ctl_be_block_io {
void (*beio_cont)(struct ctl_be_block_io *beio); /* to continue processing */
};
extern struct ctl_softc *control_softc;
static int cbb_num_threads = 14;
TUNABLE_INT("kern.cam.ctl.block.num_threads", &cbb_num_threads);
SYSCTL_NODE(_kern_cam_ctl, OID_AUTO, block, CTLFLAG_RD, 0,
@ -2228,7 +2232,13 @@ ctl_be_block_create(struct ctl_be_block_softc *softc, struct ctl_lun_req *req)
else
cbe_lun->lun_type = T_DIRECT;
be_lun->flags = CTL_BE_BLOCK_LUN_UNCONFIGURED;
cbe_lun->flags = CTL_LUN_FLAG_PRIMARY;
cbe_lun->flags = 0;
value = ctl_get_opt(&cbe_lun->options, "ha_role");
if (value != NULL) {
if (strcmp(value, "primary") == 0)
cbe_lun->flags |= CTL_LUN_FLAG_PRIMARY;
} else if (control_softc->flags & CTL_FLAG_ACTIVE_SHELF)
cbe_lun->flags |= CTL_LUN_FLAG_PRIMARY;
if (cbe_lun->lun_type == T_DIRECT) {
be_lun->size_bytes = params->lun_size_bytes;
@ -2240,10 +2250,13 @@ ctl_be_block_create(struct ctl_be_block_softc *softc, struct ctl_lun_req *req)
cbe_lun->maxlba = (be_lun->size_blocks == 0) ?
0 : (be_lun->size_blocks - 1);
retval = ctl_be_block_open(softc, be_lun, req);
if (retval != 0) {
retval = 0;
req->status = CTL_LUN_WARNING;
if ((cbe_lun->flags & CTL_LUN_FLAG_PRIMARY) ||
control_softc->ha_mode == CTL_HA_MODE_SER_ONLY) {
retval = ctl_be_block_open(softc, be_lun, req);
if (retval != 0) {
retval = 0;
req->status = CTL_LUN_WARNING;
}
}
num_threads = cbb_num_threads;
} else {
@ -2601,8 +2614,9 @@ ctl_be_block_modify(struct ctl_be_block_softc *softc, struct ctl_lun_req *req)
struct ctl_lun_modify_params *params;
struct ctl_be_block_lun *be_lun;
struct ctl_be_lun *cbe_lun;
char *value;
uint64_t oldsize;
int error;
int error, wasprim;
params = &req->reqdata.modify;
@ -2625,23 +2639,51 @@ ctl_be_block_modify(struct ctl_be_block_softc *softc, struct ctl_lun_req *req)
be_lun->params.lun_size_bytes = params->lun_size_bytes;
ctl_update_opts(&cbe_lun->options, req->num_be_args, req->kern_be_args);
oldsize = be_lun->size_blocks;
if (be_lun->vn == NULL)
error = ctl_be_block_open(softc, be_lun, req);
else if (vn_isdisk(be_lun->vn, &error))
error = ctl_be_block_modify_dev(be_lun, req);
else if (be_lun->vn->v_type == VREG)
error = ctl_be_block_modify_file(be_lun, req);
wasprim = (cbe_lun->flags & CTL_LUN_FLAG_PRIMARY);
value = ctl_get_opt(&cbe_lun->options, "ha_role");
if (value != NULL) {
if (strcmp(value, "primary") == 0)
cbe_lun->flags |= CTL_LUN_FLAG_PRIMARY;
else
cbe_lun->flags &= ~CTL_LUN_FLAG_PRIMARY;
} else if (control_softc->flags & CTL_FLAG_ACTIVE_SHELF)
cbe_lun->flags |= CTL_LUN_FLAG_PRIMARY;
else
error = EINVAL;
cbe_lun->flags &= ~CTL_LUN_FLAG_PRIMARY;
if (wasprim != (cbe_lun->flags & CTL_LUN_FLAG_PRIMARY)) {
if (cbe_lun->flags & CTL_LUN_FLAG_PRIMARY)
ctl_lun_primary(cbe_lun);
else
ctl_lun_secondary(cbe_lun);
}
oldsize = be_lun->size_blocks;
if ((cbe_lun->flags & CTL_LUN_FLAG_PRIMARY) ||
control_softc->ha_mode == CTL_HA_MODE_SER_ONLY) {
if (be_lun->vn == NULL)
error = ctl_be_block_open(softc, be_lun, req);
else if (vn_isdisk(be_lun->vn, &error))
error = ctl_be_block_modify_dev(be_lun, req);
else if (be_lun->vn->v_type == VREG)
error = ctl_be_block_modify_file(be_lun, req);
else
error = EINVAL;
if ((cbe_lun->flags & CTL_LUN_FLAG_OFFLINE) &&
be_lun->vn != NULL) {
cbe_lun->flags &= ~CTL_LUN_FLAG_OFFLINE;
ctl_lun_online(cbe_lun);
}
} else {
if (be_lun->vn != NULL) {
cbe_lun->flags |= CTL_LUN_FLAG_OFFLINE;
ctl_lun_offline(cbe_lun);
pause("CTL LUN offline", hz / 8); // XXX
error = ctl_be_block_close(be_lun);
} else
error = 0;
}
if (be_lun->size_blocks != oldsize)
ctl_lun_capacity_changed(cbe_lun);
if ((cbe_lun->flags & CTL_LUN_FLAG_OFFLINE) &&
be_lun->vn != NULL) {
cbe_lun->flags &= ~CTL_LUN_FLAG_OFFLINE;
ctl_lun_online(cbe_lun);
}
/* Tell the user the exact size we ended up using */
params->lun_size_bytes = be_lun->size_bytes;

36
sys/cam/ctl/ctl_backend_ramdisk.c
View File

@ -56,14 +56,18 @@ __FBSDID("$FreeBSD$");
#include <sys/conf.h>
#include <sys/ioccom.h>
#include <sys/module.h>
#include <sys/sysctl.h>
#include <cam/scsi/scsi_all.h>
#include <cam/scsi/scsi_da.h>
#include <cam/ctl/ctl_io.h>
#include <cam/ctl/ctl.h>
#include <cam/ctl/ctl_util.h>
#include <cam/ctl/ctl_backend.h>
#include <cam/ctl/ctl_debug.h>
#include <cam/ctl/ctl_ioctl.h>
#include <cam/ctl/ctl_ha.h>
#include <cam/ctl/ctl_private.h>
#include <cam/ctl/ctl_error.h>
typedef enum {
@ -101,6 +105,7 @@ struct ctl_be_ramdisk_softc {
};
static struct ctl_be_ramdisk_softc rd_softc;
extern struct ctl_softc *control_softc;
int ctl_backend_ramdisk_init(void);
void ctl_backend_ramdisk_shutdown(void);
@ -546,7 +551,13 @@ ctl_backend_ramdisk_create(struct ctl_be_ramdisk_softc *softc,
else
cbe_lun->lun_type = T_DIRECT;
be_lun->flags = CTL_BE_RAMDISK_LUN_UNCONFIGURED;
cbe_lun->flags = CTL_LUN_FLAG_PRIMARY;
cbe_lun->flags = 0;
value = ctl_get_opt(&cbe_lun->options, "ha_role");
if (value != NULL) {
if (strcmp(value, "primary") == 0)
cbe_lun->flags |= CTL_LUN_FLAG_PRIMARY;
} else if (control_softc->flags & CTL_FLAG_ACTIVE_SHELF)
cbe_lun->flags |= CTL_LUN_FLAG_PRIMARY;
if (cbe_lun->lun_type == T_DIRECT) {
if (params->blocksize_bytes != 0)
@ -717,7 +728,9 @@ ctl_backend_ramdisk_modify(struct ctl_be_ramdisk_softc *softc,
struct ctl_be_ramdisk_lun *be_lun;
struct ctl_be_lun *cbe_lun;
struct ctl_lun_modify_params *params;
char *value;
uint32_t blocksize;
int wasprim;
params = &req->reqdata.modify;
@ -739,15 +752,32 @@ ctl_backend_ramdisk_modify(struct ctl_be_ramdisk_softc *softc,
if (params->lun_size_bytes != 0)
be_lun->params.lun_size_bytes = params->lun_size_bytes;
ctl_update_opts(&cbe_lun->options, req->num_be_args, req->kern_be_args);
blocksize = be_lun->cbe_lun.blocksize;
wasprim = (cbe_lun->flags & CTL_LUN_FLAG_PRIMARY);
value = ctl_get_opt(&cbe_lun->options, "ha_role");
if (value != NULL) {
if (strcmp(value, "primary") == 0)
cbe_lun->flags |= CTL_LUN_FLAG_PRIMARY;
else
cbe_lun->flags &= ~CTL_LUN_FLAG_PRIMARY;
} else if (control_softc->flags & CTL_FLAG_ACTIVE_SHELF)
cbe_lun->flags |= CTL_LUN_FLAG_PRIMARY;
else
cbe_lun->flags &= ~CTL_LUN_FLAG_PRIMARY;
if (wasprim != (cbe_lun->flags & CTL_LUN_FLAG_PRIMARY)) {
if (cbe_lun->flags & CTL_LUN_FLAG_PRIMARY)
ctl_lun_primary(cbe_lun);
else
ctl_lun_secondary(cbe_lun);
}
blocksize = be_lun->cbe_lun.blocksize;
if (be_lun->params.lun_size_bytes < blocksize) {
snprintf(req->error_str, sizeof(req->error_str),
"%s: LUN size %ju < blocksize %u", __func__,
be_lun->params.lun_size_bytes, blocksize);
goto bailout_error;
}
be_lun->size_blocks = be_lun->params.lun_size_bytes / blocksize;
be_lun->size_bytes = be_lun->size_blocks * blocksize;
be_lun->cbe_lun.maxlba = be_lun->size_blocks - 1;

126
sys/cam/ctl/ctl_cmd_table.c
View File

@ -69,8 +69,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5e[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -81,8 +80,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5e[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -93,8 +91,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5e[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -105,8 +102,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5e[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -123,8 +119,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5f[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -135,8 +130,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5f[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -147,8 +141,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5f[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -159,8 +152,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5f[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -171,8 +163,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5f[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -183,8 +174,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5f[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -195,8 +185,7 @@ const struct ctl_cmd_entry ctl_cmd_table_5f[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -339,7 +328,7 @@ const struct ctl_cmd_entry ctl_cmd_table_84[32] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -436,7 +425,6 @@ const struct ctl_cmd_entry ctl_cmd_table_9e[32] =
{ctl_read_capacity_16, CTL_SERIDX_RD_CAP, CTL_CMD_FLAG_OK_ON_SLUN |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_READCAP,
@ -493,8 +481,8 @@ const struct ctl_cmd_entry ctl_cmd_table_a3[32] =
{ctl_report_tagret_port_groups, CTL_SERIDX_INQ, CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_CMD_FLAG_OK_ON_UNAVAIL |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -507,8 +495,8 @@ const struct ctl_cmd_entry ctl_cmd_table_a3[32] =
{ctl_report_supported_opcodes, CTL_SERIDX_INQ, CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_CMD_FLAG_OK_ON_UNAVAIL |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -518,8 +506,8 @@ const struct ctl_cmd_entry ctl_cmd_table_a3[32] =
{ctl_report_supported_tmf, CTL_SERIDX_INQ, CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_CMD_FLAG_OK_ON_UNAVAIL |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -532,8 +520,8 @@ const struct ctl_cmd_entry ctl_cmd_table_a3[32] =
{ctl_report_timestamp, CTL_SERIDX_INQ, CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_CMD_FLAG_OK_ON_UNAVAIL |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -563,8 +551,8 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
CTL_CMD_FLAG_NO_SENSE |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_CMD_FLAG_OK_ON_UNAVAIL |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE, 6, {0x01, 0, 0, 0xff, 0x07}},
@ -624,8 +612,8 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
CTL_CMD_FLAG_NO_SENSE |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_CMD_FLAG_OK_ON_UNAVAIL |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE, 6, {0xe1, 0xff, 0xff, 0xff, 0x07}},
@ -640,8 +628,7 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
{ctl_mode_select, CTL_SERIDX_MD_SEL, CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT,
CTL_LUN_PAT_NONE, 6, {0x11, 0, 0, 0xff, 0x07}},
@ -650,8 +637,7 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT,
CTL_LUN_PAT_NONE, 6, {0, 0, 0, 0, 0x07}},
@ -660,8 +646,7 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_NONE,
CTL_LUN_PAT_NONE, 6, {0, 0, 0, 0, 0x07}},
@ -675,8 +660,7 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
{ctl_mode_sense, CTL_SERIDX_MD_SNS, CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_WRESV,
CTL_LUN_PAT_NONE, 6, {0x08, 0xff, 0xff, 0xff, 0x07}},
@ -685,7 +669,6 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
{ctl_start_stop, CTL_SERIDX_START, CTL_CMD_FLAG_OK_ON_SLUN |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_FLAG_DATA_NONE |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE, 6, {0x01, 0, 0, 0x03, 0x07}},
@ -721,7 +704,6 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
{ctl_read_capacity, CTL_SERIDX_RD_CAP, CTL_CMD_FLAG_OK_ON_SLUN|
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_READCAP, 10, {0, 0, 0, 0, 0, 0, 0, 0, 0x07}},
@ -812,7 +794,7 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
{ctl_write_buffer, CTL_SERIDX_MD_SEL, CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT,
CTL_LUN_PAT_NONE,
10, {0x1f, 0, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x07}},
@ -821,7 +803,7 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
{ctl_read_buffer, CTL_SERIDX_MD_SNS, CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_WRESV,
CTL_LUN_PAT_NONE,
@ -911,8 +893,7 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
{ctl_mode_select, CTL_SERIDX_MD_SEL, CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT,
CTL_LUN_PAT_NONE, 10, {0x11, 0, 0, 0, 0, 0, 0xff, 0xff, 0x07} },
@ -921,8 +902,7 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT,
CTL_LUN_PAT_NONE, 10, {0x02, 0, 0xff, 0, 0, 0, 0xff, 0xff, 0x07} },
@ -931,8 +911,7 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_OUT,
CTL_LUN_PAT_NONE, 10, {0x02, 0, 0xff, 0, 0, 0, 0xff, 0xff, 0x07} },
@ -946,8 +925,7 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
{ctl_mode_sense, CTL_SERIDX_MD_SNS, CTL_CMD_FLAG_OK_ON_BOTH |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_WRESV,
CTL_LUN_PAT_NONE, 10, {0x18, 0xff, 0xff, 0, 0, 0, 0xff, 0xff, 0x07} },
@ -1199,8 +1177,8 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
CTL_CMD_FLAG_NO_SENSE |
CTL_CMD_FLAG_OK_ON_STOPPED |
CTL_CMD_FLAG_OK_ON_INOPERABLE |
CTL_CMD_FLAG_OK_ON_OFFLINE |
CTL_CMD_FLAG_OK_ON_SECONDARY |
CTL_CMD_FLAG_OK_ON_STANDBY |
CTL_CMD_FLAG_OK_ON_UNAVAIL |
CTL_FLAG_DATA_IN |
CTL_CMD_FLAG_ALLOW_ON_PR_RESV,
CTL_LUN_PAT_NONE,
@ -1315,33 +1293,17 @@ const struct ctl_cmd_entry ctl_cmd_table[256] =
/* BF VOLUME SET OUT */
{NULL, CTL_SERIDX_INVLD, CTL_CMD_FLAG_NONE, CTL_LUN_PAT_NONE},
/* C0 - ISC_SEND_MSG_SHORT */
//{NULL, CTL_SERIDX_INVLD, CTL_CMD_FLAG_NONE},
{ctl_isc, CTL_SERIDX_READ, CTL_CMD_FLAG_OK_ON_PROC | CTL_FLAG_DATA_NONE,
CTL_LUN_PAT_NONE,
16, {0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}},
/* C0 */
{NULL, CTL_SERIDX_INVLD, CTL_CMD_FLAG_NONE, CTL_LUN_PAT_NONE},
/* C1 - ISC_SEND_MSG */
//{NULL, CTL_SERIDX_INVLD, CTL_CMD_FLAG_NONE},
{ctl_isc, CTL_SERIDX_READ, CTL_CMD_FLAG_OK_ON_PROC | CTL_FLAG_DATA_OUT,
CTL_LUN_PAT_NONE,
16, {0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}},
/* C1 */
{NULL, CTL_SERIDX_INVLD, CTL_CMD_FLAG_NONE, CTL_LUN_PAT_NONE},
/* C2 - ISC_WRITE */
//{NULL, CTL_SERIDX_INVLD, CTL_CMD_FLAG_NONE},
{ctl_isc, CTL_SERIDX_READ, CTL_CMD_FLAG_OK_ON_PROC | CTL_FLAG_DATA_OUT,
CTL_LUN_PAT_NONE,
16, {0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}},
/* C2 */
{NULL, CTL_SERIDX_INVLD, CTL_CMD_FLAG_NONE, CTL_LUN_PAT_NONE},
/* C3 - ISC_READ */
//{NULL, CTL_SERIDX_INVLD, CTL_CMD_FLAG_NONE},
{ctl_isc, CTL_SERIDX_READ, CTL_CMD_FLAG_OK_ON_PROC | CTL_FLAG_DATA_IN,
CTL_LUN_PAT_NONE,
16, {0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}},
/* C3 */
{NULL, CTL_SERIDX_INVLD, CTL_CMD_FLAG_NONE, CTL_LUN_PAT_NONE},
/* C4 */
{NULL, CTL_SERIDX_INVLD, CTL_CMD_FLAG_NONE, CTL_LUN_PAT_NONE},

24
sys/cam/ctl/ctl_error.c
View File

@ -722,6 +722,18 @@ ctl_set_illegal_pr_release(struct ctl_scsiio *ctsio)
SSD_ELEM_NONE);
}
void
ctl_set_lun_transit(struct ctl_scsiio *ctsio)
{
/* "Logical unit not ready, asymmetric access state transition" */
ctl_set_sense(ctsio,
/*current_error*/ 1,
/*sense_key*/ SSD_KEY_NOT_READY,
/*asc*/ 0x04,
/*ascq*/ 0x0a,
SSD_ELEM_NONE);
}
void
ctl_set_lun_standby(struct ctl_scsiio *ctsio)
{
@ -734,6 +746,18 @@ ctl_set_lun_standby(struct ctl_scsiio *ctsio)
SSD_ELEM_NONE);
}
void
ctl_set_lun_unavail(struct ctl_scsiio *ctsio)
{
/* "Logical unit not ready, target port in unavailable state" */
ctl_set_sense(ctsio,
/*current_error*/ 1,
/*sense_key*/ SSD_KEY_NOT_READY,
/*asc*/ 0x04,
/*ascq*/ 0x0c,
SSD_ELEM_NONE);
}
void
ctl_set_medium_format_corrupted(struct ctl_scsiio *ctsio)
{

2
sys/cam/ctl/ctl_error.h
View File

@ -67,7 +67,9 @@ void ctl_set_invalid_opcode(struct ctl_scsiio *ctsio);
void ctl_set_param_len_error(struct ctl_scsiio *ctsio);
void ctl_set_already_locked(struct ctl_scsiio *ctsio);
void ctl_set_unsupported_lun(struct ctl_scsiio *ctsio);
void ctl_set_lun_transit(struct ctl_scsiio *ctsio);
void ctl_set_lun_standby(struct ctl_scsiio *ctsio);
void ctl_set_lun_unavail(struct ctl_scsiio *ctsio);
void ctl_set_internal_failure(struct ctl_scsiio *ctsio, int sks_valid,
uint16_t retry_count);
void ctl_set_medium_error(struct ctl_scsiio *ctsio);

35
sys/cam/ctl/ctl_frontend.c
View File

@ -140,6 +140,7 @@ int
ctl_port_register(struct ctl_port *port)
{
struct ctl_softc *softc = control_softc;
struct ctl_port *tport, *nport;
void *pool;
int port_num;
int retval;
@ -149,10 +150,13 @@ ctl_port_register(struct ctl_port *port)
KASSERT(softc != NULL, ("CTL is not initialized"));
mtx_lock(&softc->ctl_lock);
port_num = ctl_ffz(softc->ctl_port_mask, CTL_MAX_PORTS);
if ((port_num == -1)
|| (ctl_set_mask(softc->ctl_port_mask, port_num) == -1)) {
port->targ_port = -1;
if (port->targ_port >= 0)
port_num = port->targ_port;
else
port_num = ctl_ffz(softc->ctl_port_mask,
softc->port_min, softc->port_max);
if ((port_num < 0) ||
(ctl_set_mask(softc->ctl_port_mask, port_num) < 0)) {
mtx_unlock(&softc->ctl_lock);
return (1);
}
@ -195,10 +199,17 @@ ctl_port_register(struct ctl_port *port)
STAILQ_INIT(&port->options);
mtx_lock(&softc->ctl_lock);
port->targ_port = port_num + softc->port_offset;
port->targ_port = port_num;
STAILQ_INSERT_TAIL(&port->frontend->port_list, port, fe_links);
STAILQ_INSERT_TAIL(&softc->port_list, port, links);
softc->ctl_ports[port_num] = port;
for (tport = NULL, nport = STAILQ_FIRST(&softc->port_list);
nport != NULL && nport->targ_port < port_num;
tport = nport, nport = STAILQ_NEXT(tport, links)) {
}
if (tport)
STAILQ_INSERT_AFTER(&softc->port_list, tport, port, links);
else
STAILQ_INSERT_HEAD(&softc->port_list, port, links);
softc->ctl_ports[port->targ_port] = port;
mtx_unlock(&softc->ctl_lock);
return (retval);
@ -209,7 +220,7 @@ ctl_port_deregister(struct ctl_port *port)
{
struct ctl_softc *softc = control_softc;
struct ctl_io_pool *pool;
int port_num, retval, i;
int retval, i;
retval = 0;
@ -224,10 +235,8 @@ ctl_port_deregister(struct ctl_port *port)
STAILQ_REMOVE(&softc->port_list, port, ctl_port, links);
STAILQ_REMOVE(&port->frontend->port_list, port, ctl_port, fe_links);
softc->num_ports--;
port_num = (port->targ_port < CTL_MAX_PORTS) ? port->targ_port :
port->targ_port - CTL_MAX_PORTS;
ctl_clear_mask(softc->ctl_port_mask, port_num);
softc->ctl_ports[port_num] = NULL;
ctl_clear_mask(softc->ctl_port_mask, port->targ_port);
softc->ctl_ports[port->targ_port] = NULL;
mtx_unlock(&softc->ctl_lock);
ctl_pool_free(pool);
@ -321,6 +330,7 @@ ctl_port_online(struct ctl_port *port)
port->port_online(port->onoff_arg);
/* XXX KDM need a lock here? */
port->status |= CTL_PORT_STATUS_ONLINE;
ctl_isc_announce_port(port);
}
void
@ -347,6 +357,7 @@ ctl_port_offline(struct ctl_port *port)
}
/* XXX KDM need a lock here? */
port->status &= ~CTL_PORT_STATUS_ONLINE;
ctl_isc_announce_port(port);
}
/*

1
sys/cam/ctl/ctl_frontend_cam_sim.c
View File

@ -157,6 +157,7 @@ cfcs_init(void)
/* XXX These should probably be fetched from CTL. */
port->max_targets = 1;
port->max_target_id = 15;
port->targ_port = -1;
retval = ctl_port_register(port);
if (retval != 0) {

1
sys/cam/ctl/ctl_frontend_ioctl.c
View File

@ -93,6 +93,7 @@ cfi_init(void)
port->fe_done = cfi_done;
port->max_targets = 1;
port->max_target_id = 0;
port->targ_port = -1;
port->max_initiators = 1;
if (ctl_port_register(port) != 0) {

1
sys/cam/ctl/ctl_frontend_iscsi.c
View File

@ -2033,6 +2033,7 @@ cfiscsi_ioctl_port_create(struct ctl_req *req)
/* XXX These should probably be fetched from CTL. */
port->max_targets = 1;
port->max_target_id = 15;
port->targ_port = -1;
port->options = opts;
STAILQ_INIT(&opts);

958
sys/cam/ctl/ctl_ha.c Normal file
View File

@ -0,0 +1,958 @@
/*-
* Copyright (c) 2015 Alexander Motin <mav@FreeBSD.org>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer,
* without modification, immediately at the beginning of the file.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
* IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
* OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
* IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
* NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
* THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/kernel.h>
#include <sys/kthread.h>
#include <sys/types.h>
#include <sys/limits.h>
#include <sys/lock.h>
#include <sys/module.h>
#include <sys/mutex.h>
#include <sys/condvar.h>
#include <sys/malloc.h>
#include <sys/mbuf.h>
#include <sys/proc.h>
#include <sys/conf.h>
#include <sys/queue.h>
#include <sys/sysctl.h>
#include <sys/socket.h>
#include <sys/socketvar.h>
#include <sys/uio.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <vm/uma.h>
#include <cam/cam.h>
#include <cam/scsi/scsi_all.h>
#include <cam/scsi/scsi_da.h>
#include <cam/ctl/ctl_io.h>
#include <cam/ctl/ctl.h>
#include <cam/ctl/ctl_frontend.h>
#include <cam/ctl/ctl_util.h>
#include <cam/ctl/ctl_backend.h>
#include <cam/ctl/ctl_ioctl.h>
#include <cam/ctl/ctl_ha.h>
#include <cam/ctl/ctl_private.h>
#include <cam/ctl/ctl_debug.h>
#include <cam/ctl/ctl_error.h>
#if (__FreeBSD_version < 1100000)
struct mbufq {
struct mbuf *head;
struct mbuf *tail;
};
static void
mbufq_init(struct mbufq *q, int limit)
{
q->head = q->tail = NULL;
}
static void
mbufq_drain(struct mbufq *q)
{
struct mbuf *m;
while ((m = q->head) != NULL) {
q->head = m->m_nextpkt;
m_freem(m);
}
q->tail = NULL;
}
static struct mbuf *
mbufq_dequeue(struct mbufq *q)
{
struct mbuf *m;
m = q->head;
if (m) {
if (q->tail == m)
q->tail = NULL;
q->head = m->m_nextpkt;
m->m_nextpkt = NULL;
}
return (m);
}
static void
mbufq_enqueue(struct mbufq *q, struct mbuf *m)
{
m->m_nextpkt = NULL;
if (q->tail)
q->tail->m_nextpkt = m;
else
q->head = m;
q->tail = m;
}
static u_int
sbavail(struct sockbuf *sb)
{
return (sb->sb_cc);
}
#if (__FreeBSD_version < 1000000)
#define mtodo(m, o) ((void *)(((m)->m_data) + (o)))
#endif
#endif
struct ha_msg_wire {
uint32_t channel;
uint32_t length;
};
struct ha_dt_msg_wire {
ctl_ha_dt_cmd command;
uint32_t size;
uint8_t *local;
uint8_t *remote;
};
struct ha_softc {
struct ctl_softc *ha_ctl_softc;
ctl_evt_handler ha_handler[CTL_HA_CHAN_MAX];
char ha_peer[128];
struct sockaddr_in ha_peer_in;
struct socket *ha_lso;
struct socket *ha_so;
struct mbufq ha_sendq;
struct mbuf *ha_sending;
struct mtx ha_lock;
int ha_connect;
int ha_listen;
int ha_connected;
int ha_receiving;
int ha_wakeup;
int ha_disconnect;
TAILQ_HEAD(, ctl_ha_dt_req) ha_dts;
} ha_softc;
extern struct ctl_softc *control_softc;
static void
ctl_ha_conn_wake(struct ha_softc *softc)
{
mtx_lock(&softc->ha_lock);
softc->ha_wakeup = 1;
mtx_unlock(&softc->ha_lock);
wakeup(&softc->ha_wakeup);
}
static int
ctl_ha_lupcall(struct socket *so, void *arg, int waitflag)
{
struct ha_softc *softc = arg;
ctl_ha_conn_wake(softc);
return (SU_OK);
}
static int
ctl_ha_rupcall(struct socket *so, void *arg, int waitflag)
{
struct ha_softc *softc = arg;
wakeup(&softc->ha_receiving);
return (SU_OK);
}
static int
ctl_ha_supcall(struct socket *so, void *arg, int waitflag)
{
struct ha_softc *softc = arg;
ctl_ha_conn_wake(softc);
return (SU_OK);
}
static void
ctl_ha_evt(struct ha_softc *softc, ctl_ha_channel ch, ctl_ha_event evt,
int param)
{
int i;
if (ch < CTL_HA_CHAN_MAX) {
if (softc->ha_handler[ch])
softc->ha_handler[ch](ch, evt, param);
return;
}
for (i = 0; i < CTL_HA_CHAN_MAX; i++) {
if (softc->ha_handler[i])
softc->ha_handler[i](i, evt, param);
}
}
static void
ctl_ha_close(struct ha_softc *softc)
{
struct socket *so = softc->ha_so;
int report = 0;
if (softc->ha_connected || softc->ha_disconnect) {
softc->ha_connected = 0;
mbufq_drain(&softc->ha_sendq);
m_freem(softc->ha_sending);
softc->ha_sending = NULL;
report = 1;
}
if (so) {
SOCKBUF_LOCK(&so->so_rcv);
soupcall_clear(so, SO_RCV);
while (softc->ha_receiving) {
wakeup(&softc->ha_receiving);
msleep(&softc->ha_receiving, SOCKBUF_MTX(&so->so_rcv),
0, "ha_rx exit", 0);
}
SOCKBUF_UNLOCK(&so->so_rcv);
SOCKBUF_LOCK(&so->so_snd);
soupcall_clear(so, SO_SND);
SOCKBUF_UNLOCK(&so->so_snd);
softc->ha_so = NULL;
if (softc->ha_connect)
pause("reconnect", hz / 2);
soclose(so);
}
if (report) {
ctl_ha_evt(softc, CTL_HA_CHAN_MAX, CTL_HA_EVT_LINK_CHANGE,
(softc->ha_connect || softc->ha_listen) ?
CTL_HA_LINK_UNKNOWN : CTL_HA_LINK_OFFLINE);
}
}
static void
ctl_ha_lclose(struct ha_softc *softc)
{
if (softc->ha_lso) {
SOCKBUF_LOCK(&softc->ha_lso->so_rcv);
soupcall_clear(softc->ha_lso, SO_RCV);
SOCKBUF_UNLOCK(&softc->ha_lso->so_rcv);
soclose(softc->ha_lso);
softc->ha_lso = NULL;
}
}
static void
ctl_ha_rx_thread(void *arg)
{
struct ha_softc *softc = arg;
struct socket *so = softc->ha_so;
struct ha_msg_wire wire_hdr;
struct uio uio;
struct iovec iov;
int error, flags, next;
bzero(&wire_hdr, sizeof(wire_hdr));
while (1) {
if (wire_hdr.length > 0)
next = wire_hdr.length;
else
next = sizeof(wire_hdr);
SOCKBUF_LOCK(&so->so_rcv);
while (sbavail(&so->so_rcv) < next) {
if (softc->ha_connected == 0 || so->so_error ||
(so->so_rcv.sb_state & SBS_CANTRCVMORE)) {
goto errout;
}
so->so_rcv.sb_lowat = next;
msleep(&softc->ha_receiving, SOCKBUF_MTX(&so->so_rcv),
0, "-", 0);
}
SOCKBUF_UNLOCK(&so->so_rcv);
if (wire_hdr.length == 0) {
iov.iov_base = &wire_hdr;
iov.iov_len = sizeof(wire_hdr);
uio.uio_iov = &iov;
uio.uio_iovcnt = 1;
uio.uio_rw = UIO_READ;
uio.uio_segflg = UIO_SYSSPACE;
uio.uio_td = curthread;
uio.uio_resid = sizeof(wire_hdr);
flags = MSG_DONTWAIT;
error = soreceive(softc->ha_so, NULL, &uio, NULL,
NULL, &flags);
if (error != 0) {
printf("%s: header receive error %d\n",
__func__, error);
SOCKBUF_LOCK(&so->so_rcv);
goto errout;
}
} else {
ctl_ha_evt(softc, wire_hdr.channel,
CTL_HA_EVT_MSG_RECV, wire_hdr.length);
wire_hdr.length = 0;
}
}
errout:
softc->ha_receiving = 0;
wakeup(&softc->ha_receiving);
SOCKBUF_UNLOCK(&so->so_rcv);
ctl_ha_conn_wake(softc);
kthread_exit();
}
static void
ctl_ha_send(struct ha_softc *softc)
{
struct socket *so = softc->ha_so;
int error;
while (1) {
if (softc->ha_sending == NULL) {
mtx_lock(&softc->ha_lock);
softc->ha_sending = mbufq_dequeue(&softc->ha_sendq);
mtx_unlock(&softc->ha_lock);
if (softc->ha_sending == NULL) {
so->so_snd.sb_lowat = so->so_snd.sb_hiwat + 1;
break;
}
}
SOCKBUF_LOCK(&so->so_snd);
if (sbspace(&so->so_snd) < softc->ha_sending->m_pkthdr.len) {
so->so_snd.sb_lowat = softc->ha_sending->m_pkthdr.len;
SOCKBUF_UNLOCK(&so->so_snd);
break;
}
SOCKBUF_UNLOCK(&so->so_snd);
error = sosend(softc->ha_so, NULL, NULL, softc->ha_sending,
NULL, MSG_DONTWAIT, curthread);
softc->ha_sending = NULL;
if (error != 0) {
printf("%s: sosend() error %d\n", __func__, error);
return;
}
};
}
static void
ctl_ha_sock_setup(struct ha_softc *softc)
{
struct sockopt opt;
struct socket *so = softc->ha_so;
int error, val;
val = 1024 * 1024;
error = soreserve(so, val, val);
if (error)
printf("%s: soreserve failed %d\n", __func__, error);
SOCKBUF_LOCK(&so->so_rcv);
so->so_rcv.sb_lowat = sizeof(struct ha_msg_wire);
soupcall_set(so, SO_RCV, ctl_ha_rupcall, softc);
SOCKBUF_UNLOCK(&so->so_rcv);
SOCKBUF_LOCK(&so->so_snd);
so->so_snd.sb_lowat = sizeof(struct ha_msg_wire);
soupcall_set(so, SO_SND, ctl_ha_supcall, softc);
SOCKBUF_UNLOCK(&so->so_snd);
bzero(&opt, sizeof(struct sockopt));
opt.sopt_dir = SOPT_SET;
opt.sopt_level = SOL_SOCKET;
opt.sopt_name = SO_KEEPALIVE;
opt.sopt_val = &val;
opt.sopt_valsize = sizeof(val);
val = 1;
error = sosetopt(so, &opt);
if (error)
printf("%s: KEEPALIVE setting failed %d\n", __func__, error);
opt.sopt_level = IPPROTO_TCP;
opt.sopt_name = TCP_NODELAY;
val = 1;
error = sosetopt(so, &opt);
if (error)
printf("%s: NODELAY setting failed %d\n", __func__, error);
opt.sopt_name = TCP_KEEPINIT;
val = 3;
error = sosetopt(so, &opt);
if (error)
printf("%s: KEEPINIT setting failed %d\n", __func__, error);
opt.sopt_name = TCP_KEEPIDLE;
val = 1;
error = sosetopt(so, &opt);
if (error)
printf("%s: KEEPIDLE setting failed %d\n", __func__, error);
opt.sopt_name = TCP_KEEPINTVL;
val = 1;
error = sosetopt(so, &opt);
if (error)
printf("%s: KEEPINTVL setting failed %d\n", __func__, error);
opt.sopt_name = TCP_KEEPCNT;
val = 5;
error = sosetopt(so, &opt);
if (error)
printf("%s: KEEPCNT setting failed %d\n", __func__, error);
}
static int
ctl_ha_connect(struct ha_softc *softc)
{
struct thread *td = curthread;
struct socket *so;
int error;
/* Create the socket */
error = socreate(PF_INET, &so, SOCK_STREAM,
IPPROTO_TCP, td->td_ucred, td);
if (error != 0) {
printf("%s: socreate() error %d\n", __func__, error);
return (error);
}
softc->ha_so = so;
ctl_ha_sock_setup(softc);
error = soconnect(so, (struct sockaddr *)&softc->ha_peer_in, td);
if (error != 0) {
printf("%s: soconnect() error %d\n", __func__, error);
goto out;
}
return (0);
out:
ctl_ha_close(softc);
return (error);
}
static int
ctl_ha_accept(struct ha_softc *softc)
{
struct socket *so;
struct sockaddr *sap;
int error;
ACCEPT_LOCK();
if (softc->ha_lso->so_rcv.sb_state & SBS_CANTRCVMORE)
softc->ha_lso->so_error = ECONNABORTED;
if (softc->ha_lso->so_error) {
error = softc->ha_lso->so_error;
softc->ha_lso->so_error = 0;
ACCEPT_UNLOCK();
printf("%s: socket error %d\n", __func__, error);
goto out;
}
so = TAILQ_FIRST(&softc->ha_lso->so_comp);
if (so == NULL) {
ACCEPT_UNLOCK();
return (EWOULDBLOCK);
}
KASSERT(!(so->so_qstate & SQ_INCOMP), ("accept1: so SQ_INCOMP"));
KASSERT(so->so_qstate & SQ_COMP, ("accept1: so not SQ_COMP"));
/*
* Before changing the flags on the socket, we have to bump the
* reference count. Otherwise, if the protocol calls sofree(),
* the socket will be released due to a zero refcount.
*/
SOCK_LOCK(so); /* soref() and so_state update */
soref(so); /* file descriptor reference */
TAILQ_REMOVE(&softc->ha_lso->so_comp, so, so_list);
softc->ha_lso->so_qlen--;
so->so_state |= SS_NBIO;
so->so_qstate &= ~SQ_COMP;
so->so_head = NULL;
SOCK_UNLOCK(so);
ACCEPT_UNLOCK();
sap = NULL;
error = soaccept(so, &sap);
if (error != 0) {
printf("%s: soaccept() error %d\n", __func__, error);
if (sap != NULL)
free(sap, M_SONAME);
goto out;
}
if (sap != NULL)
free(sap, M_SONAME);
softc->ha_so = so;
ctl_ha_sock_setup(softc);
return (0);
out:
ctl_ha_lclose(softc);
return (error);
}
static int
ctl_ha_listen(struct ha_softc *softc)
{
struct thread *td = curthread;
struct sockopt opt;
int error, val;
/* Create the socket */
if (softc->ha_lso == NULL) {
error = socreate(PF_INET, &softc->ha_lso, SOCK_STREAM,
IPPROTO_TCP, td->td_ucred, td);
if (error != 0) {
printf("%s: socreate() error %d\n", __func__, error);
return (error);
}
bzero(&opt, sizeof(struct sockopt));
opt.sopt_dir = SOPT_SET;
opt.sopt_level = SOL_SOCKET;
opt.sopt_name = SO_REUSEADDR;
opt.sopt_val = &val;
opt.sopt_valsize = sizeof(val);
val = 1;
error = sosetopt(softc->ha_lso, &opt);
if (error) {
printf("%s: REUSEADDR setting failed %d\n",
__func__, error);
}
SOCKBUF_LOCK(&softc->ha_lso->so_rcv);
soupcall_set(softc->ha_lso, SO_RCV, ctl_ha_lupcall, softc);
SOCKBUF_UNLOCK(&softc->ha_lso->so_rcv);
}
error = sobind(softc->ha_lso, (struct sockaddr *)&softc->ha_peer_in, td);
if (error != 0) {
printf("%s: sobind() error %d\n", __func__, error);
goto out;
}
error = solisten(softc->ha_lso, 1, td);
if (error != 0) {
printf("%s: solisten() error %d\n", __func__, error);
goto out;
}
return (0);
out:
ctl_ha_lclose(softc);
return (error);
}
static void
ctl_ha_conn_thread(void *arg)
{
struct ha_softc *softc = arg;
int error;
while (1) {
if (softc->ha_disconnect) {
ctl_ha_close(softc);
ctl_ha_lclose(softc);
softc->ha_disconnect = 0;
} else if (softc->ha_so != NULL &&
(softc->ha_so->so_error ||
softc->ha_so->so_rcv.sb_state & SBS_CANTRCVMORE))
ctl_ha_close(softc);
if (softc->ha_so == NULL) {
if (softc->ha_lso != NULL)
ctl_ha_accept(softc);
else if (softc->ha_listen)
ctl_ha_listen(softc);
else if (softc->ha_connect)
ctl_ha_connect(softc);
}
if (softc->ha_so != NULL) {
if (softc->ha_connected == 0 &&
softc->ha_so->so_error == 0 &&
(softc->ha_so->so_state & SS_ISCONNECTING) == 0) {
softc->ha_connected = 1;
ctl_ha_evt(softc, CTL_HA_CHAN_MAX,
CTL_HA_EVT_LINK_CHANGE,
CTL_HA_LINK_ONLINE);
softc->ha_receiving = 1;
error = kproc_kthread_add(ctl_ha_rx_thread,
softc, &softc->ha_ctl_softc->ctl_proc,
NULL, 0, 0, "ctl", "ha_rx");
if (error != 0) {
printf("Error creating CTL HA rx thread!\n");
softc->ha_receiving = 0;
softc->ha_disconnect = 1;
}
}
ctl_ha_send(softc);
}
mtx_lock(&softc->ha_lock);
if (softc->ha_so != NULL &&
(softc->ha_so->so_error ||
softc->ha_so->so_rcv.sb_state & SBS_CANTRCVMORE))
;
else if (!softc->ha_wakeup)
msleep(&softc->ha_wakeup, &softc->ha_lock, 0, "-", hz);
softc->ha_wakeup = 0;
mtx_unlock(&softc->ha_lock);
}
}
static int
ctl_ha_peer_sysctl(SYSCTL_HANDLER_ARGS)
{
struct ha_softc *softc = (struct ha_softc *)arg1;
struct sockaddr_in *sa;
int error, b1, b2, b3, b4, p, num;
error = sysctl_handle_string(oidp, softc->ha_peer,
sizeof(softc->ha_peer), req);
if ((error != 0) || (req->newptr == NULL))
return (error);
sa = &softc->ha_peer_in;
mtx_lock(&softc->ha_lock);
if ((num = sscanf(softc->ha_peer, "connect %d.%d.%d.%d:%d",
&b1, &b2, &b3, &b4, &p)) >= 4) {
softc->ha_connect = 1;
softc->ha_listen = 0;
} else if ((num = sscanf(softc->ha_peer, "listen %d.%d.%d.%d:%d",
&b1, &b2, &b3, &b4, &p)) >= 4) {
softc->ha_connect = 0;
softc->ha_listen = 1;
} else {
softc->ha_connect = 0;
softc->ha_listen = 0;
if (softc->ha_peer[0] != 0)
error = EINVAL;
}
if (softc->ha_connect || softc->ha_listen) {
memset(sa, 0, sizeof(*sa));
sa->sin_len = sizeof(struct sockaddr_in);
sa->sin_family = AF_INET;
sa->sin_port = htons((num >= 5) ? p : 999);
sa->sin_addr.s_addr =
htonl((b1 << 24) + (b2 << 16) + (b3 << 8) + b4);
}
softc->ha_disconnect = 1;
softc->ha_wakeup = 1;
mtx_unlock(&softc->ha_lock);
wakeup(&softc->ha_wakeup);
return (error);
}
ctl_ha_status
ctl_ha_msg_register(ctl_ha_channel channel, ctl_evt_handler handler)
{
struct ha_softc *softc = &ha_softc;
KASSERT(channel < CTL_HA_CHAN_MAX,
("Wrong CTL HA channel %d", channel));
softc->ha_handler[channel] = handler;
return (CTL_HA_STATUS_SUCCESS);
}
ctl_ha_status
ctl_ha_msg_deregister(ctl_ha_channel channel)
{
struct ha_softc *softc = &ha_softc;
KASSERT(channel < CTL_HA_CHAN_MAX,
("Wrong CTL HA channel %d", channel));
softc->ha_handler[channel] = NULL;
return (CTL_HA_STATUS_SUCCESS);
}
/*
* Receive a message of the specified size.
*/
ctl_ha_status
ctl_ha_msg_recv(ctl_ha_channel channel, void *addr, size_t len,
int wait)
{
struct ha_softc *softc = &ha_softc;
struct uio uio;
struct iovec iov;
int error, flags;
if (!softc->ha_connected)
return (CTL_HA_STATUS_DISCONNECT);
iov.iov_base = addr;
iov.iov_len = len;
uio.uio_iov = &iov;
uio.uio_iovcnt = 1;
uio.uio_rw = UIO_READ;
uio.uio_segflg = UIO_SYSSPACE;
uio.uio_td = curthread;
uio.uio_resid = len;
flags = wait ? 0 : MSG_DONTWAIT;
error = soreceive(softc->ha_so, NULL, &uio, NULL, NULL, &flags);
if (error == 0)
return (CTL_HA_STATUS_SUCCESS);
/* Consider all errors fatal for HA sanity. */
mtx_lock(&softc->ha_lock);
if (softc->ha_connected) {
softc->ha_disconnect = 1;
softc->ha_wakeup = 1;
wakeup(&softc->ha_wakeup);
}
mtx_unlock(&softc->ha_lock);
return (CTL_HA_STATUS_ERROR);
}
/*
* Send a message of the specified size.
*/
ctl_ha_status
ctl_ha_msg_send2(ctl_ha_channel channel, const void *addr, size_t len,
const void *addr2, size_t len2, int wait)
{
struct ha_softc *softc = &ha_softc;
struct mbuf *mb, *newmb;
struct ha_msg_wire hdr;
size_t copylen, off;
if (!softc->ha_connected)
return (CTL_HA_STATUS_DISCONNECT);
newmb = m_getm2(NULL, sizeof(hdr) + len + len2, wait, MT_DATA,
M_PKTHDR);
if (newmb == NULL) {
/* Consider all errors fatal for HA sanity. */
mtx_lock(&softc->ha_lock);
if (softc->ha_connected) {
softc->ha_disconnect = 1;
softc->ha_wakeup = 1;
wakeup(&softc->ha_wakeup);
}
mtx_unlock(&softc->ha_lock);
printf("%s: Can't allocate mbuf chain\n", __func__);
return (CTL_HA_STATUS_ERROR);
}
hdr.channel = channel;
hdr.length = len + len2;
mb = newmb;
memcpy(mtodo(mb, 0), &hdr, sizeof(hdr));
mb->m_len += sizeof(hdr);
off = 0;
for (; mb != NULL && off < len; mb = mb->m_next) {
copylen = min(M_TRAILINGSPACE(mb), len - off);
memcpy(mtodo(mb, mb->m_len), (const char *)addr + off, copylen);
mb->m_len += copylen;
off += copylen;
if (off == len)
break;
}
KASSERT(off == len, ("%s: off (%zu) != len (%zu)", __func__,
off, len));
off = 0;
for (; mb != NULL && off < len2; mb = mb->m_next) {
copylen = min(M_TRAILINGSPACE(mb), len2 - off);
memcpy(mtodo(mb, mb->m_len), (const char *)addr2 + off, copylen);
mb->m_len += copylen;
off += copylen;
}
KASSERT(off == len2, ("%s: off (%zu) != len2 (%zu)", __func__,
off, len2));
newmb->m_pkthdr.len = sizeof(hdr) + len + len2;
mtx_lock(&softc->ha_lock);
if (!softc->ha_connected) {
mtx_unlock(&softc->ha_lock);
m_freem(newmb);
return (CTL_HA_STATUS_DISCONNECT);
}
mbufq_enqueue(&softc->ha_sendq, newmb);
softc->ha_wakeup = 1;
mtx_unlock(&softc->ha_lock);
wakeup(&softc->ha_wakeup);
return (CTL_HA_STATUS_SUCCESS);
}
ctl_ha_status
ctl_ha_msg_send(ctl_ha_channel channel, const void *addr, size_t len,
int wait)
{
return (ctl_ha_msg_send2(channel, addr, len, NULL, 0, wait));
}
/*
* Allocate a data transfer request structure.
*/
struct ctl_ha_dt_req *
ctl_dt_req_alloc(void)
{
return (malloc(sizeof(struct ctl_ha_dt_req), M_CTL, M_WAITOK | M_ZERO));
}
/*
* Free a data transfer request structure.
*/
void
ctl_dt_req_free(struct ctl_ha_dt_req *req)
{
free(req, M_CTL);
}
/*
* Issue a DMA request for a single buffer.
*/
ctl_ha_status
ctl_dt_single(struct ctl_ha_dt_req *req)
{
struct ha_softc *softc = &ha_softc;
struct ha_dt_msg_wire wire_dt;
ctl_ha_status status;
wire_dt.command = req->command;
wire_dt.size = req->size;
wire_dt.local = req->local;
wire_dt.remote = req->remote;
if (req->command == CTL_HA_DT_CMD_READ && req->callback != NULL) {
mtx_lock(&softc->ha_lock);
TAILQ_INSERT_TAIL(&softc->ha_dts, req, links);
mtx_unlock(&softc->ha_lock);
ctl_ha_msg_send(CTL_HA_CHAN_DATA, &wire_dt, sizeof(wire_dt),
M_WAITOK);
return (CTL_HA_STATUS_WAIT);
}
if (req->command == CTL_HA_DT_CMD_READ) {
status = ctl_ha_msg_send(CTL_HA_CHAN_DATA, &wire_dt,
sizeof(wire_dt), M_WAITOK);
} else {
status = ctl_ha_msg_send2(CTL_HA_CHAN_DATA, &wire_dt,
sizeof(wire_dt), req->local, req->size, M_WAITOK);
}
return (status);
}
static void
ctl_dt_event_handler(ctl_ha_channel channel, ctl_ha_event event, int param)
{
struct ha_softc *softc = &ha_softc;
struct ctl_ha_dt_req *req;
ctl_ha_status isc_status;
if (event == CTL_HA_EVT_MSG_RECV) {
struct ha_dt_msg_wire wire_dt;
uint8_t *tmp;
int size;
size = min(sizeof(wire_dt), param);
isc_status = ctl_ha_msg_recv(CTL_HA_CHAN_DATA, &wire_dt,
size, M_WAITOK);
if (isc_status != CTL_HA_STATUS_SUCCESS) {
printf("%s: Error receiving message: %d\n",
__func__, isc_status);
return;
}
if (wire_dt.command == CTL_HA_DT_CMD_READ) {
wire_dt.command = CTL_HA_DT_CMD_WRITE;
tmp = wire_dt.local;
wire_dt.local = wire_dt.remote;
wire_dt.remote = tmp;
ctl_ha_msg_send2(CTL_HA_CHAN_DATA, &wire_dt,
sizeof(wire_dt), wire_dt.local, wire_dt.size,
M_WAITOK);
} else if (wire_dt.command == CTL_HA_DT_CMD_WRITE) {
isc_status = ctl_ha_msg_recv(CTL_HA_CHAN_DATA,
wire_dt.remote, wire_dt.size, M_WAITOK);
mtx_lock(&softc->ha_lock);
TAILQ_FOREACH(req, &softc->ha_dts, links) {
if (req->local == wire_dt.remote) {
TAILQ_REMOVE(&softc->ha_dts, req, links);
break;
}
}
mtx_unlock(&softc->ha_lock);
if (req) {
req->ret = isc_status;
req->callback(req);
}
}
} else if (event == CTL_HA_EVT_LINK_CHANGE) {
CTL_DEBUG_PRINT(("%s: Link state change to %d\n", __func__,
param));
if (param != CTL_HA_LINK_ONLINE) {
mtx_lock(&softc->ha_lock);
while ((req = TAILQ_FIRST(&softc->ha_dts)) != NULL) {
TAILQ_REMOVE(&softc->ha_dts, req, links);
mtx_unlock(&softc->ha_lock);
req->ret = CTL_HA_STATUS_DISCONNECT;
req->callback(req);
mtx_lock(&softc->ha_lock);
}
mtx_unlock(&softc->ha_lock);
}
} else {
printf("%s: Unknown event %d\n", __func__, event);
}
}
ctl_ha_status
ctl_ha_msg_init(struct ctl_softc *ctl_softc)
{
struct ha_softc *softc = &ha_softc;
int error;
softc->ha_ctl_softc = ctl_softc;
mtx_init(&softc->ha_lock, "CTL HA mutex", NULL, MTX_DEF);
mbufq_init(&softc->ha_sendq, INT_MAX);
TAILQ_INIT(&softc->ha_dts);
error = kproc_kthread_add(ctl_ha_conn_thread, softc,
&ctl_softc->ctl_proc, NULL, 0, 0, "ctl", "ha_tx");
if (error != 0) {
printf("error creating CTL HA connection thread!\n");
mtx_destroy(&softc->ha_lock);
return (CTL_HA_STATUS_ERROR);
}
SYSCTL_ADD_PROC(&ctl_softc->sysctl_ctx,
SYSCTL_CHILDREN(ctl_softc->sysctl_tree),
OID_AUTO, "ha_peer", CTLTYPE_STRING | CTLFLAG_RWTUN,
softc, 0, ctl_ha_peer_sysctl, "A", "HA peer connection method");
if (ctl_ha_msg_register(CTL_HA_CHAN_DATA, ctl_dt_event_handler)
!= CTL_HA_STATUS_SUCCESS) {
printf("%s: ctl_ha_msg_register failed.\n", __func__);
}
return (CTL_HA_STATUS_SUCCESS);
};
ctl_ha_status
ctl_ha_msg_shutdown(struct ctl_softc *ctl_softc)
{
struct ha_softc *softc = &ha_softc;
if (ctl_ha_msg_deregister(CTL_HA_CHAN_DATA) != CTL_HA_STATUS_SUCCESS) {
printf("%s: ctl_ha_msg_deregister failed.\n", __func__);
}
mtx_destroy(&softc->ha_lock);
return (CTL_HA_STATUS_SUCCESS);
};

196
sys/cam/ctl/ctl_ha.h
View File

@ -1,6 +1,7 @@
/*-
* Copyright (c) 2003-2009 Silicon Graphics International Corp.
* Copyright (c) 2011 Spectra Logic Corporation
* Copyright (c) 2015 Alexander Motin <mav@FreeBSD.org>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
@ -38,80 +39,27 @@
/*
* CTL High Availability Modes:
*
* CTL_HA_MODE_ACT_STBY: One side is in Active state and processing commands,
* the other side is in Standby state, returning errors.
* CTL_HA_MODE_SER_ONLY: Commands are serialized to the other side. Write
* mirroring and read re-direction are assumed to
* happen in the back end.
* CTL_HA_MODE_XFER: Commands are serialized and data is transferred
* for write mirroring and read re-direction.
* CTL_HA_MODE_ACT_STBY: Commands are serialized to the master side.
* No media access commands on slave side (Standby).
* CTL_HA_MODE_SER_ONLY: Commands are serialized to the master side.
* Media can be accessed on both sides.
* CTL_HA_MODE_XFER: Commands and data are forwarded to the
* master side for execution.
*/
typedef enum {
CTL_HA_MODE_ACT_STBY,
CTL_HA_MODE_SER_ONLY,
CTL_HA_MODE_XFER
} ctl_ha_mode;
/*
* This is a stubbed out High Availability interface. It assumes two nodes
* staying in sync.
*
* The reason this interface is here, and stubbed out, is that CTL was
* originally written with support for Copan's (now SGI) high availability
* framework. That framework was not released by SGI, and would not have
* been generally applicable to FreeBSD anyway.
*
* The idea here is to show the kind of API that would need to be in place
* in a HA framework to work with CTL's HA hooks. This API is very close
* to the Copan/SGI API, so that the code using it could stay in place
* as-is.
*
* So, in summary, this is a shell without real substance, and much more
* work would be needed to actually make HA work. The implementation
* inside CTL will also need to change to fit the eventual implementation.
* The additional pieces we would need are:
*
* - HA "Supervisor" framework that can startup the components of the
* system, and initiate failover (i.e. active/active to single mode)
* and failback (single to active/active mode) state transitions.
* This framework would be able to recognize when an event happens
* that requires it to initiate state transitions in the components it
* manages.
*
* - HA communication framework. This framework should have the following
* features:
* - Separate channels for separate system components. The CTL
* instance on one node should communicate with the CTL instance
* on another node.
* - Short message passing. These messages would be fixed length, so
* they could be preallocated and easily passed between the nodes.
* i.e. conceptually like an ethernet packet.
* - DMA/large buffer capability. This would require some negotiation
* with the other node to define the destination. It could
* allow for "push" (i.e. initiated by the requesting node) DMA or
* "pull" (i.e. initiated by the target controller) DMA or both.
* - Communication channel status change notification.
* - HA capability in other portions of the storage stack. Having two CTL
* instances communicate is just one part of an overall HA solution.
* State needs to be synchronized at multiple levels of the system in
* order for failover to actually work. For instance, if CTL is using a
* file on a ZFS filesystem as its backing store, the ZFS array state
* should be synchronized with the other node, so that the other node
* can immediately take over if the node that is primary for a particular
* array fails.
*/
/*
* Communication channel IDs for various system components. This is to
* make sure one CTL instance talks with another, one ZFS instance talks
* with another, etc.
*/
typedef enum {
CTL_HA_CHAN_NONE,
CTL_HA_CHAN_CTL,
CTL_HA_CHAN_ZFS,
CTL_HA_CHAN_DATA,
CTL_HA_CHAN_MAX
} ctl_ha_channel;
@ -120,18 +68,12 @@ typedef enum {
* HA communication subsystem.
*
* CTL_HA_EVT_MSG_RECV: Message received by the other node.
* CTL_HA_EVT_MSG_SENT: Message sent to the other node.
* CTL_HA_EVT_DISCONNECT: Communication channel disconnected.
* CTL_HA_EVT_DMA_SENT: DMA successfully sent to other node (push).
* CTL_HA_EVT_DMA_RECEIVED: DMA successfully received by other node (pull).
* CTL_HA_EVT_LINK_CHANGE: Communication channel status changed.
*/
typedef enum {
CTL_HA_EVT_NONE,
CTL_HA_EVT_MSG_RECV,
CTL_HA_EVT_MSG_SENT,
CTL_HA_EVT_DISCONNECT,
CTL_HA_EVT_DMA_SENT,
CTL_HA_EVT_DMA_RECEIVED,
CTL_HA_EVT_LINK_CHANGE,
CTL_HA_EVT_MAX
} ctl_ha_event;
@ -145,12 +87,6 @@ typedef enum {
CTL_HA_STATUS_MAX
} ctl_ha_status;
typedef enum {
CTL_HA_DATA_CTL,
CTL_HA_DATA_ZFS,
CTL_HA_DATA_MAX
} ctl_ha_dtid;
typedef enum {
CTL_HA_DT_CMD_READ,
CTL_HA_DT_CMD_WRITE,
@ -164,110 +100,40 @@ struct ctl_ha_dt_req {
ctl_ha_dt_cmd command;
void *context;
ctl_ha_dt_cb callback;
ctl_ha_dtid id;
int ret;
uint32_t size;
uint8_t *local;
uint8_t *remote;
TAILQ_ENTRY(ctl_ha_dt_req) links;
};
struct ctl_softc;
ctl_ha_status ctl_ha_msg_init(struct ctl_softc *softc);
ctl_ha_status ctl_ha_msg_shutdown(struct ctl_softc *softc);
typedef void (*ctl_evt_handler)(ctl_ha_channel channel, ctl_ha_event event,
int param);
void ctl_ha_register_evthandler(ctl_ha_channel channel,
ctl_evt_handler handler);
static inline ctl_ha_status
ctl_ha_msg_create(ctl_ha_channel channel, ctl_evt_handler handler)
{
return (CTL_HA_STATUS_SUCCESS);
}
ctl_ha_status ctl_ha_msg_register(ctl_ha_channel channel,
ctl_evt_handler handler);
ctl_ha_status ctl_ha_msg_recv(ctl_ha_channel channel, void *addr,
size_t len, int wait);
ctl_ha_status ctl_ha_msg_send(ctl_ha_channel channel, const void *addr,
size_t len, int wait);
ctl_ha_status ctl_ha_msg_send2(ctl_ha_channel channel, const void *addr,
size_t len, const void *addr2, size_t len2, int wait);
ctl_ha_status ctl_ha_msg_deregister(ctl_ha_channel channel);
/*
* Receive a message of the specified size.
*/
static inline ctl_ha_status
ctl_ha_msg_recv(ctl_ha_channel channel, void *buffer, unsigned int size,
int wait)
{
return (CTL_HA_STATUS_SUCCESS);
}
/*
* Send a message of the specified size.
*/
static inline ctl_ha_status
ctl_ha_msg_send(ctl_ha_channel channel, void *buffer, unsigned int size,
int wait)
{
return (CTL_HA_STATUS_SUCCESS);
}
/*
* Allocate a data transfer request structure.
*/
static inline struct ctl_ha_dt_req *
ctl_dt_req_alloc(void)
{
return (NULL);
}
/*
* Free a data transfer request structure.
*/
static inline void
ctl_dt_req_free(struct ctl_ha_dt_req *req)
{
return;
}
/*
* Issue a DMA request for a single buffer.
*/
static inline ctl_ha_status
ctl_dt_single(struct ctl_ha_dt_req *req)
{
return (CTL_HA_STATUS_WAIT);
}
/*
* SINGLE: One node
* HA: Two nodes (Active/Active implied)
* SLAVE/MASTER: The component can set these flags to indicate which side
* is in control. It has no effect on the HA framework.
*/
typedef enum {
CTL_HA_STATE_UNKNOWN = 0x00,
CTL_HA_STATE_SINGLE = 0x01,
CTL_HA_STATE_HA = 0x02,
CTL_HA_STATE_MASK = 0x0F,
CTL_HA_STATE_SLAVE = 0x10,
CTL_HA_STATE_MASTER = 0x20
} ctl_ha_state;
struct ctl_ha_dt_req * ctl_dt_req_alloc(void);
void ctl_dt_req_free(struct ctl_ha_dt_req *req);
ctl_ha_status ctl_dt_single(struct ctl_ha_dt_req *req);
typedef enum {
CTL_HA_COMP_STATUS_OK,
CTL_HA_COMP_STATUS_FAILED,
CTL_HA_COMP_STATUS_ERROR
} ctl_ha_comp_status;
struct ctl_ha_component;
typedef ctl_ha_comp_status (*ctl_hacmp_init_t)(struct ctl_ha_component *);
typedef ctl_ha_comp_status (*ctl_hacmp_start_t)(struct ctl_ha_component *,
ctl_ha_state);
struct ctl_ha_component {
char *name;
ctl_ha_state state;
ctl_ha_comp_status status;
ctl_hacmp_init_t init;
ctl_hacmp_start_t start;
ctl_hacmp_init_t quiesce;
};
#define CTL_HA_STATE_IS_SINGLE(state) ((state & CTL_HA_STATE_MASK) == \
CTL_HA_STATE_SINGLE)
#define CTL_HA_STATE_IS_HA(state) ((state & CTL_HA_STATE_MASK) == \
CTL_HA_STATE_HA)
CTL_HA_LINK_OFFLINE = 0x00,
CTL_HA_LINK_UNKNOWN = 0x01,
CTL_HA_LINK_ONLINE = 0x02
} ctl_ha_link_state;
#endif /* _CTL_HA_H_ */

81
sys/cam/ctl/ctl_io.h
View File

@ -92,13 +92,11 @@ typedef enum {
CTL_FLAG_EDPTR_SGLIST = 0x00000010, /* ext_data_ptr is S/G list */
CTL_FLAG_DO_AUTOSENSE = 0x00000020, /* grab sense info */
CTL_FLAG_USER_REQ = 0x00000040, /* request came from userland */
CTL_FLAG_CONTROL_DEV = 0x00000080, /* processor device */
CTL_FLAG_ALLOCATED = 0x00000100, /* data space allocated */
CTL_FLAG_BLOCKED = 0x00000200, /* on the blocked queue */
CTL_FLAG_ABORT_STATUS = 0x00000400, /* return TASK ABORTED status */
CTL_FLAG_ABORT = 0x00000800, /* this I/O should be aborted */
CTL_FLAG_DMA_INPROG = 0x00001000, /* DMA in progress */
CTL_FLAG_NO_DATASYNC = 0x00002000, /* don't cache flush data */
CTL_FLAG_DELAY_DONE = 0x00004000, /* delay injection done */
CTL_FLAG_INT_COPY = 0x00008000, /* internal copy, no done call*/
CTL_FLAG_SENT_2OTHER_SC = 0x00010000,
@ -108,9 +106,6 @@ typedef enum {
addresses, not virtual ones*/
CTL_FLAG_IO_CONT = 0x00100000, /* Continue I/O instead of
completing */
CTL_FLAG_AUTO_MIRROR = 0x00200000, /* Automatically use memory
from the RC cache mirrored
address area. */
#if 0
CTL_FLAG_ALREADY_DONE = 0x00200000 /* I/O already completed */
#endif
@ -118,14 +113,8 @@ typedef enum {
CTL_FLAG_DMA_QUEUED = 0x00800000, /* DMA queued but not started*/
CTL_FLAG_STATUS_QUEUED = 0x01000000, /* Status queued but not sent*/
CTL_FLAG_REDIR_DONE = 0x02000000, /* Redirection has already
been done. */
CTL_FLAG_FAILOVER = 0x04000000, /* Killed by a failover */
CTL_FLAG_IO_ACTIVE = 0x08000000, /* I/O active on this SC */
CTL_FLAG_RDMA_MASK = CTL_FLAG_NO_DATASYNC | CTL_FLAG_BUS_ADDR |
CTL_FLAG_AUTO_MIRROR | CTL_FLAG_REDIR_DONE,
/* Flags we care about for
remote DMA */
CTL_FLAG_STATUS_SENT = 0x10000000 /* Status sent by datamove */
} ctl_io_flags;
@ -203,15 +192,16 @@ typedef enum {
CTL_MSG_BAD_JUJU,
CTL_MSG_MANAGE_TASKS,
CTL_MSG_PERS_ACTION,
CTL_MSG_SYNC_FE,
CTL_MSG_DATAMOVE,
CTL_MSG_DATAMOVE_DONE
CTL_MSG_DATAMOVE_DONE,
CTL_MSG_UA, /* Set/clear UA on secondary. */
CTL_MSG_PORT_SYNC, /* Information about port. */
CTL_MSG_LUN_SYNC, /* Information about LUN. */
CTL_MSG_FAILOVER /* Fake, never sent though the wire */
} ctl_msg_type;
struct ctl_scsiio;
#define CTL_NUM_SG_ENTRIES 9
struct ctl_io_hdr {
uint32_t version; /* interface version XXX */
ctl_io_type io_type; /* task I/O, SCSI I/O, etc. */
@ -237,10 +227,8 @@ struct ctl_io_hdr {
union ctl_io *serializing_sc;
void *pool; /* I/O pool */
union ctl_priv ctl_private[CTL_NUM_PRIV];/* CTL private area */
struct ctl_sg_entry remote_sglist[CTL_NUM_SG_ENTRIES];
struct ctl_sg_entry remote_dma_sglist[CTL_NUM_SG_ENTRIES];
struct ctl_sg_entry local_sglist[CTL_NUM_SG_ENTRIES];
struct ctl_sg_entry local_dma_sglist[CTL_NUM_SG_ENTRIES];
struct ctl_sg_entry *remote_sglist;
struct ctl_sg_entry *local_sglist;
STAILQ_ENTRY(ctl_io_hdr) links; /* linked list pointer */
TAILQ_ENTRY(ctl_io_hdr) ooa_links;
TAILQ_ENTRY(ctl_io_hdr) blocked_links;
@ -386,10 +374,10 @@ struct ctl_ha_msg_hdr {
union ctl_io *serializing_sc;
struct ctl_nexus nexus; /* Initiator, port, target, lun */
uint32_t status; /* transaction status */
TAILQ_ENTRY(ctl_ha_msg_hdr) links;
};
#define CTL_HA_MAX_SG_ENTRIES 16
#define CTL_HA_DATAMOVE_SEGMENT 131072
/*
* Used for CTL_MSG_PERS_ACTION.
@ -399,6 +387,16 @@ struct ctl_ha_msg_pr {
struct ctl_pr_info pr_info;
};
/*
* Used for CTL_MSG_UA.
*/
struct ctl_ha_msg_ua {
struct ctl_ha_msg_hdr hdr;
int ua_all;
int ua_set;
int ua_type;
};
/*
* The S/G handling here is a little different than the standard ctl_scsiio
* structure, because we can't pass data by reference in between controllers.
@ -431,17 +429,18 @@ struct ctl_ha_msg_dt {
*/
struct ctl_ha_msg_scsi {
struct ctl_ha_msg_hdr hdr;
uint8_t cdb[CTL_MAX_CDBLEN]; /* CDB */
uint32_t tag_num; /* tag number */
ctl_tag_type tag_type; /* simple, ordered, etc. */
uint8_t cdb[CTL_MAX_CDBLEN]; /* CDB */
uint8_t cdb_len; /* CDB length */
uint8_t scsi_status; /* SCSI status byte */
struct scsi_sense_data sense_data; /* sense data */
uint8_t sense_len; /* Returned sense length */
uint8_t sense_residual; /* sense residual length */
uint32_t residual; /* data residual length */
uint32_t fetd_status; /* trans status, set by FETD,
0 = good*/
struct ctl_lba_len lbalen; /* used for stats */
struct scsi_sense_data sense_data; /* sense data */
};
/*
@ -454,12 +453,50 @@ struct ctl_ha_msg_task {
ctl_tag_type tag_type; /* simple, ordered, etc. */
};
/*
* Used for CTL_MSG_PORT_SYNC.
*/
struct ctl_ha_msg_port {
struct ctl_ha_msg_hdr hdr;
int port_type;
int physical_port;
int virtual_port;
int status;
int name_len;
int lun_map_len;
int port_devid_len;
int target_devid_len;
uint8_t data[];
};
/*
* Used for CTL_MSG_LUN_SYNC.
*/
struct ctl_ha_msg_lun {
struct ctl_ha_msg_hdr hdr;
int flags;
unsigned int pr_generation;
uint32_t pr_res_idx;
uint8_t pr_res_type;
int lun_devid_len;
int pr_key_count;
uint8_t data[];
};
struct ctl_ha_msg_lun_pr_key {
uint32_t pr_iid;
uint64_t pr_key;
};
union ctl_ha_msg {
struct ctl_ha_msg_hdr hdr;
struct ctl_ha_msg_task task;
struct ctl_ha_msg_scsi scsi;
struct ctl_ha_msg_dt dt;
struct ctl_ha_msg_pr pr;
struct ctl_ha_msg_ua ua;
struct ctl_ha_msg_port port;
struct ctl_ha_msg_lun lun;
};

22
sys/cam/ctl/ctl_private.h
View File

@ -106,8 +106,8 @@ typedef enum {
CTL_CMD_FLAG_OK_ON_BOTH = 0x0300,
CTL_CMD_FLAG_OK_ON_STOPPED = 0x0400,
CTL_CMD_FLAG_OK_ON_INOPERABLE = 0x0800,
CTL_CMD_FLAG_OK_ON_OFFLINE = 0x1000,
CTL_CMD_FLAG_OK_ON_SECONDARY = 0x2000,
CTL_CMD_FLAG_OK_ON_STANDBY = 0x1000,
CTL_CMD_FLAG_OK_ON_UNAVAIL = 0x2000,
CTL_CMD_FLAG_ALLOW_ON_PR_RESV = 0x4000,
CTL_CMD_FLAG_SA5 = 0x8000
} ctl_cmd_flags;
@ -157,7 +157,8 @@ typedef enum {
CTL_LUN_PR_RESERVED = 0x100,
CTL_LUN_PRIMARY_SC = 0x200,
CTL_LUN_SENSE_DESC = 0x400,
CTL_LUN_READONLY = 0x800
CTL_LUN_READONLY = 0x800,
CTL_LUN_PEER_SC_PRIMARY = 0x1000
} ctl_lun_flags;
typedef enum {
@ -398,7 +399,7 @@ struct ctl_lun {
struct ctl_lun_io_stats stats;
uint32_t res_idx;
unsigned int PRGeneration;
uint64_t *pr_keys[2 * CTL_MAX_PORTS];
uint64_t *pr_keys[CTL_MAX_PORTS];
int pr_key_count;
uint32_t pr_res_idx;
uint8_t res_type;
@ -434,11 +435,13 @@ struct ctl_softc {
ctl_gen_flags flags;
ctl_ha_mode ha_mode;
int ha_id;
int ha_state;
int is_single;
int port_offset;
int persis_offset;
int inquiry_pq_no_lun;
ctl_ha_link_state ha_link;
int port_min;
int port_max;
int port_cnt;
int init_min;
int init_max;
struct sysctl_ctx_list sysctl_ctx;
struct sysctl_oid *sysctl_tree;
void *othersc_pool;
@ -469,8 +472,6 @@ struct ctl_softc {
extern const struct ctl_cmd_entry ctl_cmd_table[256];
uint32_t ctl_get_initindex(struct ctl_nexus *nexus);
uint32_t ctl_get_resindex(struct ctl_nexus *nexus);
uint32_t ctl_port_idx(int port_num);
int ctl_lun_map_init(struct ctl_port *port);
int ctl_lun_map_deinit(struct ctl_port *port);
int ctl_lun_map_set(struct ctl_port *port, uint32_t plun, uint32_t glun);
@ -508,7 +509,6 @@ int ctl_report_tagret_port_groups(struct ctl_scsiio *ctsio);
int ctl_report_supported_opcodes(struct ctl_scsiio *ctsio);
int ctl_report_supported_tmf(struct ctl_scsiio *ctsio);
int ctl_report_timestamp(struct ctl_scsiio *ctsio);
int ctl_isc(struct ctl_scsiio *ctsio);
int ctl_get_lba_status(struct ctl_scsiio *ctsio);
void ctl_tpc_init(struct ctl_softc *softc);

20
sys/cam/ctl/ctl_tpc.c
View File

@ -534,7 +534,7 @@ ctl_receive_copy_status_lid1(struct ctl_scsiio *ctsio)
list_id = cdb->list_identifier;
mtx_lock(&lun->lun_lock);
list = tpc_find_list(lun, list_id,
ctl_get_resindex(&ctsio->io_hdr.nexus));
ctl_get_initindex(&ctsio->io_hdr.nexus));
if (list == NULL) {
mtx_unlock(&lun->lun_lock);
ctl_set_invalid_field(ctsio, /*sks_valid*/ 1,
@ -616,7 +616,7 @@ ctl_receive_copy_failure_details(struct ctl_scsiio *ctsio)
list_id = cdb->list_identifier;
mtx_lock(&lun->lun_lock);
list = tpc_find_list(lun, list_id,
ctl_get_resindex(&ctsio->io_hdr.nexus));
ctl_get_initindex(&ctsio->io_hdr.nexus));
if (list == NULL || !list->completed) {
mtx_unlock(&lun->lun_lock);
ctl_set_invalid_field(ctsio, /*sks_valid*/ 1,
@ -688,7 +688,7 @@ ctl_receive_copy_status_lid4(struct ctl_scsiio *ctsio)
list_id = scsi_4btoul(cdb->list_identifier);
mtx_lock(&lun->lun_lock);
list = tpc_find_list(lun, list_id,
ctl_get_resindex(&ctsio->io_hdr.nexus));
ctl_get_initindex(&ctsio->io_hdr.nexus));
if (list == NULL) {
mtx_unlock(&lun->lun_lock);
ctl_set_invalid_field(ctsio, /*sks_valid*/ 1,
@ -771,7 +771,7 @@ ctl_copy_operation_abort(struct ctl_scsiio *ctsio)
list_id = scsi_4btoul(cdb->list_identifier);
mtx_lock(&lun->lun_lock);
list = tpc_find_list(lun, list_id,
ctl_get_resindex(&ctsio->io_hdr.nexus));
ctl_get_initindex(&ctsio->io_hdr.nexus));
if (list == NULL) {
mtx_unlock(&lun->lun_lock);
ctl_set_invalid_field(ctsio, /*sks_valid*/ 1,
@ -1645,7 +1645,7 @@ ctl_extended_copy_lid1(struct ctl_scsiio *ctsio)
list->init_port = -1;
else
list->init_port = ctsio->io_hdr.nexus.targ_port;
list->init_idx = ctl_get_resindex(&ctsio->io_hdr.nexus);
list->init_idx = ctl_get_initindex(&ctsio->io_hdr.nexus);
list->list_id = data->list_identifier;
list->flags = data->flags;
list->params = ctsio->kern_data_ptr;
@ -1772,7 +1772,7 @@ ctl_extended_copy_lid4(struct ctl_scsiio *ctsio)
list->init_port = -1;
else
list->init_port = ctsio->io_hdr.nexus.targ_port;
list->init_idx = ctl_get_resindex(&ctsio->io_hdr.nexus);
list->init_idx = ctl_get_initindex(&ctsio->io_hdr.nexus);
list->list_id = scsi_4btoul(data->list_identifier);
list->flags = data->flags;
list->params = ctsio->kern_data_ptr;
@ -1890,7 +1890,7 @@ ctl_populate_token(struct ctl_scsiio *ctsio)
lun = (struct ctl_lun *)ctsio->io_hdr.ctl_private[CTL_PRIV_LUN].ptr;
softc = lun->ctl_softc;
port = softc->ctl_ports[ctl_port_idx(ctsio->io_hdr.nexus.targ_port)];
port = softc->ctl_ports[ctsio->io_hdr.nexus.targ_port];
cdb = (struct scsi_populate_token *)ctsio->cdb;
len = scsi_4btoul(cdb->length);
@ -1944,7 +1944,7 @@ ctl_populate_token(struct ctl_scsiio *ctsio)
list = malloc(sizeof(struct tpc_list), M_CTL, M_WAITOK | M_ZERO);
list->service_action = cdb->service_action;
list->init_port = ctsio->io_hdr.nexus.targ_port;
list->init_idx = ctl_get_resindex(&ctsio->io_hdr.nexus);
list->init_idx = ctl_get_initindex(&ctsio->io_hdr.nexus);
list->list_id = scsi_4btoul(cdb->list_identifier);
list->flags = data->flags;
list->ctsio = ctsio;
@ -2070,7 +2070,7 @@ ctl_write_using_token(struct ctl_scsiio *ctsio)
list = malloc(sizeof(struct tpc_list), M_CTL, M_WAITOK | M_ZERO);
list->service_action = cdb->service_action;
list->init_port = ctsio->io_hdr.nexus.targ_port;
list->init_idx = ctl_get_resindex(&ctsio->io_hdr.nexus);
list->init_idx = ctl_get_initindex(&ctsio->io_hdr.nexus);
list->list_id = scsi_4btoul(cdb->list_identifier);
list->flags = data->flags;
list->params = ctsio->kern_data_ptr;
@ -2162,7 +2162,7 @@ ctl_receive_rod_token_information(struct ctl_scsiio *ctsio)
list_id = scsi_4btoul(cdb->list_identifier);
mtx_lock(&lun->lun_lock);
list = tpc_find_list(lun, list_id,
ctl_get_resindex(&ctsio->io_hdr.nexus));
ctl_get_initindex(&ctsio->io_hdr.nexus));
if (list == NULL) {
mtx_unlock(&lun->lun_lock);
ctl_set_invalid_field(ctsio, /*sks_valid*/ 1,

8
sys/cam/ctl/ctl_tpc_local.c
View File

@ -97,11 +97,11 @@ tpcl_init(void)
port->fe_done = tpcl_done;
port->max_targets = 1;
port->max_target_id = 0;
port->targ_port = -1;
port->max_initiators = 1;
if (ctl_port_register(port) != 0)
{
printf("%s: tpc frontend registration failed\n", __func__);
if (ctl_port_register(port) != 0) {
printf("%s: ctl_port_register() failed with error\n", __func__);
return (0);
}
@ -287,7 +287,7 @@ tpcl_resolve(struct ctl_softc *softc, int init_port,
cscdid = (struct scsi_ec_cscd_id *)cscd;
mtx_lock(&softc->ctl_lock);
if (init_port >= 0)
port = softc->ctl_ports[ctl_port_idx(init_port)];
port = softc->ctl_ports[init_port];
else
port = NULL;
STAILQ_FOREACH(lun, &softc->lun_list, links) {

1
sys/cam/ctl/scsi_ctl.c
View File

@ -400,6 +400,7 @@ ctlfeasync(void *callback_arg, uint32_t code, struct cam_path *path, void *arg)
*/
port->max_targets = cpi->max_target;
port->max_target_id = cpi->max_target;
port->targ_port = -1;
/*
* XXX KDM need to figure out whether we're the master or

1
sys/conf/files
View File

@ -85,6 +85,7 @@ cam/ctl/ctl_frontend.c optional ctl
cam/ctl/ctl_frontend_cam_sim.c optional ctl
cam/ctl/ctl_frontend_ioctl.c optional ctl
cam/ctl/ctl_frontend_iscsi.c optional ctl
cam/ctl/ctl_ha.c optional ctl
cam/ctl/ctl_scsi_all.c optional ctl
cam/ctl/ctl_tpc.c optional ctl
cam/ctl/ctl_tpc_local.c optional ctl

1
sys/modules/ctl/Makefile
View File

@ -13,6 +13,7 @@ SRCS+= ctl_frontend.c
SRCS+= ctl_frontend_cam_sim.c
SRCS+= ctl_frontend_ioctl.c
SRCS+= ctl_frontend_iscsi.c
SRCS+= ctl_ha.c
SRCS+= ctl_scsi_all.c
SRCS+= ctl_tpc.c
SRCS+= ctl_tpc_local.c