63ee471b64
MD040 - Fenced code blocks should have a language specified Fixed all errors Signed-off-by: Maciej Wawryk <maciejx.wawryk@intel.com> Change-Id: Iddd307068c1047ca9a0bb12c1b0d9c88f496765f Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9272 Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com> Community-CI: Mellanox Build Bot Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Reviewed-by: Jim Harris <james.r.harris@intel.com>
332 lines
11 KiB
Markdown
332 lines
11 KiB
Markdown
# iSCSI Target {#iscsi}
|
|
|
|
## iSCSI Target Getting Started Guide {#iscsi_getting_started}
|
|
|
|
The Storage Performance Development Kit iSCSI target application is named `iscsi_tgt`.
|
|
This following section describes how to run iscsi from your cloned package.
|
|
|
|
## Prerequisites {#iscsi_prereqs}
|
|
|
|
This guide starts by assuming that you can already build the standard SPDK distribution on your
|
|
platform.
|
|
|
|
Once built, the binary will be in `build/bin`.
|
|
|
|
If you want to kill the application by using signal, make sure use the SIGTERM, then the application
|
|
will release all the shared memory resource before exit, the SIGKILL will make the shared memory
|
|
resource have no chance to be released by applications, you may need to release the resource manually.
|
|
|
|
## Introduction
|
|
|
|
The following diagram shows relations between different parts of iSCSI structure described in this
|
|
document.
|
|
|
|
|
|
|
|
### Assigning CPU Cores to the iSCSI Target {#iscsi_config_lcore}
|
|
|
|
SPDK uses the [DPDK Environment Abstraction Layer](http://dpdk.org/doc/guides/prog_guide/env_abstraction_layer.html)
|
|
to gain access to hardware resources such as huge memory pages and CPU core(s). DPDK EAL provides
|
|
functions to assign threads to specific cores.
|
|
To ensure the SPDK iSCSI target has the best performance, place the NICs and the NVMe devices on the
|
|
same NUMA node and configure the target to run on CPU cores associated with that node. The following
|
|
command line option is used to configure the SPDK iSCSI target:
|
|
|
|
~~~bash
|
|
-m 0xF000000
|
|
~~~
|
|
|
|
This is a hexadecimal bit mask of the CPU cores where the iSCSI target will start polling threads.
|
|
In this example, CPU cores 24, 25, 26 and 27 would be used.
|
|
|
|
## Configuring iSCSI Target via RPC method {#iscsi_rpc}
|
|
|
|
The iSCSI target is configured via JSON-RPC calls. See @ref jsonrpc for details.
|
|
|
|
### Portal groups
|
|
|
|
- iscsi_create_portal_group -- Add a portal group.
|
|
- iscsi_delete_portal_group -- Delete an existing portal group.
|
|
- iscsi_target_node_add_pg_ig_maps -- Add initiator group to portal group mappings to an existing iSCSI target node.
|
|
- iscsi_target_node_remove_pg_ig_maps -- Delete initiator group to portal group mappings from an existing iSCSI target node.
|
|
- iscsi_get_portal_groups -- Show information about all available portal groups.
|
|
|
|
~~~bash
|
|
/path/to/spdk/scripts/rpc.py iscsi_create_portal_group 1 10.0.0.1:3260
|
|
~~~
|
|
|
|
### Initiator groups
|
|
|
|
- iscsi_create_initiator_group -- Add an initiator group.
|
|
- iscsi_delete_initiator_group -- Delete an existing initiator group.
|
|
- iscsi_initiator_group_add_initiators -- Add initiators to an existing initiator group.
|
|
- iscsi_get_initiator_groups -- Show information about all available initiator groups.
|
|
|
|
~~~bash
|
|
/path/to/spdk/scripts/rpc.py iscsi_create_initiator_group 2 ANY 10.0.0.2/32
|
|
~~~
|
|
|
|
### Target nodes
|
|
|
|
- iscsi_create_target_node -- Add an iSCSI target node.
|
|
- iscsi_delete_target_node -- Delete an iSCSI target node.
|
|
- iscsi_target_node_add_lun -- Add a LUN to an existing iSCSI target node.
|
|
- iscsi_get_target_nodes -- Show information about all available iSCSI target nodes.
|
|
|
|
~~~bash
|
|
/path/to/spdk/scripts/rpc.py iscsi_create_target_node Target3 Target3_alias MyBdev:0 1:2 64 -d
|
|
~~~
|
|
|
|
## Configuring iSCSI Initiator {#iscsi_initiator}
|
|
|
|
The Linux initiator is open-iscsi.
|
|
|
|
Installing open-iscsi package
|
|
Fedora:
|
|
~~~bash
|
|
yum install -y iscsi-initiator-utils
|
|
~~~
|
|
|
|
Ubuntu:
|
|
~~~bash
|
|
apt-get install -y open-iscsi
|
|
~~~
|
|
|
|
### Setup
|
|
|
|
Edit /etc/iscsi/iscsid.conf
|
|
~~~bash
|
|
node.session.cmds_max = 4096
|
|
node.session.queue_depth = 128
|
|
~~~
|
|
|
|
iscsid must be restarted or receive SIGHUP for changes to take effect. To send SIGHUP, run:
|
|
~~~bash
|
|
killall -HUP iscsid
|
|
~~~
|
|
|
|
Recommended changes to /etc/sysctl.conf
|
|
~~~bash
|
|
net.ipv4.tcp_timestamps = 1
|
|
net.ipv4.tcp_sack = 0
|
|
|
|
net.ipv4.tcp_rmem = 10000000 10000000 10000000
|
|
net.ipv4.tcp_wmem = 10000000 10000000 10000000
|
|
net.ipv4.tcp_mem = 10000000 10000000 10000000
|
|
net.core.rmem_default = 524287
|
|
net.core.wmem_default = 524287
|
|
net.core.rmem_max = 524287
|
|
net.core.wmem_max = 524287
|
|
net.core.optmem_max = 524287
|
|
net.core.netdev_max_backlog = 300000
|
|
~~~
|
|
|
|
### Discovery
|
|
|
|
Assume target is at 10.0.0.1
|
|
|
|
~~~bash
|
|
iscsiadm -m discovery -t sendtargets -p 10.0.0.1
|
|
~~~
|
|
|
|
### Connect to target
|
|
|
|
~~~bash
|
|
iscsiadm -m node --login
|
|
~~~
|
|
|
|
At this point the iSCSI target should show up as SCSI disks. Check dmesg to see what
|
|
they came up as.
|
|
|
|
### Disconnect from target
|
|
|
|
~~~bash
|
|
iscsiadm -m node --logout
|
|
~~~
|
|
|
|
### Deleting target node cache
|
|
|
|
~~~bash
|
|
iscsiadm -m node -o delete
|
|
~~~
|
|
|
|
This will cause the initiator to forget all previously discovered iSCSI target nodes.
|
|
|
|
### Finding /dev/sdX nodes for iSCSI LUNs
|
|
|
|
~~~bash
|
|
iscsiadm -m session -P 3 | grep "Attached scsi disk" | awk '{print $4}'
|
|
~~~
|
|
|
|
This will show the /dev node name for each SCSI LUN in all logged in iSCSI sessions.
|
|
|
|
### Tuning
|
|
|
|
After the targets are connected, they can be tuned. For example if /dev/sdc is
|
|
an iSCSI disk then the following can be done:
|
|
Set noop to scheduler
|
|
|
|
~~~bash
|
|
echo noop > /sys/block/sdc/queue/scheduler
|
|
~~~
|
|
|
|
Disable merging/coalescing (can be useful for precise workload measurements)
|
|
|
|
~~~bash
|
|
echo "2" > /sys/block/sdc/queue/nomerges
|
|
~~~
|
|
|
|
Increase requests for block queue
|
|
|
|
~~~bash
|
|
echo "1024" > /sys/block/sdc/queue/nr_requests
|
|
~~~
|
|
|
|
### Example: Configure simple iSCSI Target with one portal and two LUNs
|
|
|
|
Assuming we have one iSCSI Target server with portal at 10.0.0.1:3200, two LUNs (Malloc0 and Malloc1),
|
|
and accepting initiators on 10.0.0.2/32, like on diagram below:
|
|
|
|
|
|
|
|
#### Configure iSCSI Target
|
|
|
|
Start iscsi_tgt application:
|
|
|
|
```bash
|
|
./build/bin/iscsi_tgt
|
|
```
|
|
|
|
Construct two 64MB Malloc block devices with 512B sector size "Malloc0" and "Malloc1":
|
|
|
|
```bash
|
|
./scripts/rpc.py bdev_malloc_create -b Malloc0 64 512
|
|
./scripts/rpc.py bdev_malloc_create -b Malloc1 64 512
|
|
```
|
|
|
|
Create new portal group with id 1, and address 10.0.0.1:3260:
|
|
|
|
```bash
|
|
./scripts/rpc.py iscsi_create_portal_group 1 10.0.0.1:3260
|
|
```
|
|
|
|
Create one initiator group with id 2 to accept any connection from 10.0.0.2/32:
|
|
|
|
```bash
|
|
./scripts/rpc.py iscsi_create_initiator_group 2 ANY 10.0.0.2/32
|
|
```
|
|
|
|
Finally construct one target using previously created bdevs as LUN0 (Malloc0) and LUN1 (Malloc1)
|
|
with a name "disk1" and alias "Data Disk1" using portal group 1 and initiator group 2.
|
|
|
|
```bash
|
|
./scripts/rpc.py iscsi_create_target_node disk1 "Data Disk1" "Malloc0:0 Malloc1:1" 1:2 64 -d
|
|
```
|
|
|
|
#### Configure initiator
|
|
|
|
Discover target
|
|
|
|
~~~bash
|
|
$ iscsiadm -m discovery -t sendtargets -p 10.0.0.1
|
|
10.0.0.1:3260,1 iqn.2016-06.io.spdk:disk1
|
|
~~~
|
|
|
|
Connect to the target
|
|
|
|
~~~bash
|
|
iscsiadm -m node --login
|
|
~~~
|
|
|
|
At this point the iSCSI target should show up as SCSI disks.
|
|
|
|
Check dmesg to see what they came up as. In this example it can look like below:
|
|
|
|
~~~bash
|
|
...
|
|
[630111.860078] scsi host68: iSCSI Initiator over TCP/IP
|
|
[630112.124743] scsi 68:0:0:0: Direct-Access INTEL Malloc disk 0001 PQ: 0 ANSI: 5
|
|
[630112.125445] sd 68:0:0:0: [sdd] 131072 512-byte logical blocks: (67.1 MB/64.0 MiB)
|
|
[630112.125468] sd 68:0:0:0: Attached scsi generic sg3 type 0
|
|
[630112.125926] sd 68:0:0:0: [sdd] Write Protect is off
|
|
[630112.125934] sd 68:0:0:0: [sdd] Mode Sense: 83 00 00 08
|
|
[630112.126049] sd 68:0:0:0: [sdd] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
|
|
[630112.126483] scsi 68:0:0:1: Direct-Access INTEL Malloc disk 0001 PQ: 0 ANSI: 5
|
|
[630112.127096] sd 68:0:0:1: Attached scsi generic sg4 type 0
|
|
[630112.127143] sd 68:0:0:1: [sde] 131072 512-byte logical blocks: (67.1 MB/64.0 MiB)
|
|
[630112.127566] sd 68:0:0:1: [sde] Write Protect is off
|
|
[630112.127573] sd 68:0:0:1: [sde] Mode Sense: 83 00 00 08
|
|
[630112.127728] sd 68:0:0:1: [sde] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
|
|
[630112.128246] sd 68:0:0:0: [sdd] Attached SCSI disk
|
|
[630112.129789] sd 68:0:0:1: [sde] Attached SCSI disk
|
|
...
|
|
~~~
|
|
|
|
You may also use simple bash command to find /dev/sdX nodes for each iSCSI LUN
|
|
in all logged iSCSI sessions:
|
|
|
|
~~~bash
|
|
$ iscsiadm -m session -P 3 | grep "Attached scsi disk" | awk '{print $4}'
|
|
sdd
|
|
sde
|
|
~~~
|
|
|
|
## iSCSI Hotplug {#iscsi_hotplug}
|
|
|
|
At the iSCSI level, we provide the following support for Hotplug:
|
|
|
|
1. bdev/nvme:
|
|
|
|
At the bdev/nvme level, we start one hotplug monitor which will call
|
|
spdk_nvme_probe() periodically to get the hotplug events. We provide the
|
|
private attach_cb and remove_cb for spdk_nvme_probe(). For the attach_cb,
|
|
we will create the block device base on the NVMe device attached, and for the
|
|
remove_cb, we will unregister the block device, which will also notify the
|
|
upper level stack (for iSCSI target, the upper level stack is scsi/lun) to
|
|
handle the hot-remove event.
|
|
|
|
2. scsi/lun:
|
|
|
|
When the LUN receive the hot-remove notification from block device layer,
|
|
the LUN will be marked as removed, and all the IOs after this point will
|
|
return with check condition status. Then the LUN starts one poller which will
|
|
wait for all the commands which have already been submitted to block device to
|
|
return back; after all the commands return back, the LUN will be deleted.
|
|
|
|
@sa spdk_nvme_probe
|
|
|
|
## iSCSI Login Redirection {#iscsi_login_redirection}
|
|
|
|
The SPDK iSCSI target application supports iSCSI login redirection feature.
|
|
|
|
A portal refers to an IP address and TCP port number pair, and a portal group
|
|
contains a set of portals. Users for the SPDK iSCSI target application configure
|
|
portals through portal groups.
|
|
|
|
To support login redirection feature, we utilize two types of portal groups,
|
|
public portal group and private portal group.
|
|
|
|
The SPDK iSCSI target application usually has a discovery portal. The discovery
|
|
portal is connected by an initiator to get a list of targets, as well as the list
|
|
of portals on which these target may be accessed, by a discovery session.
|
|
|
|
Public portal groups have their portals returned by a discovery session. Private
|
|
portal groups do not have their portals returned by a discovery session. A public
|
|
portal group may optionally have a redirect portal for non-discovery logins for
|
|
each associated target. This redirect portal must be from a private portal group.
|
|
|
|
Initiators configure portals in public portal groups as target portals. When an
|
|
initator logs in to a target through a portal in an associated public portal group,
|
|
the target sends a temporary redirection response with a redirect portal. Then the
|
|
initiator logs in to the target again through the redirect portal.
|
|
|
|
Users set a portal group to public or private at creation using the
|
|
`iscsi_create_portal_group` RPC, associate portal groups with a target using the
|
|
`iscsi_create_target_node` RPC or the `iscsi_target_node_add_pg_ig_maps` RPC,
|
|
specify a up-to-date redirect portal in a public portal group for a target using
|
|
the `iscsi_target_node_set_redirect` RPC, and terminate the corresponding connections
|
|
by asynchronous logout request using the `iscsi_target_node_request_logout` RPC.
|
|
|
|
Typically users will use the login redirection feature in scale out iSCSI target
|
|
system, which runs multiple SPDK iSCSI target applications.
|