numam-spdk/doc/nvme/multi_process.md
GangCao 7f819108bd doc: add the documentation for NVMe Multi-Process feature
Change-Id: Iace3e4f1804be19f46044ee3b2530e49c8adf4d8
Signed-off-by: GangCao <gang.cao@intel.com>
2017-03-17 14:44:03 -07:00

2.9 KiB

NVMe Multi Process

This capability enables the SPDK NVMe driver to support multiple processes accessing the same NVMe device. The NVMe driver allocates critical structures from shared memory, so that each process can map that memory and create its own queue pairs or share the admin queue. There is a limited number of I/O queue pairs per NVMe controller.

The primary motivation for this feature is to support management tools that can attach to long running applications, perform some maintenance work or gather information, and then detach.

Configuration

DPDK EAL allows different types of processes to be spawned, each with different permissions on the hugepage memory used by the applications.

There are two types of processes:

  1. a primary process which initializes the shared memory and has full privileges and
  2. a secondary process which can attach to the primary process by mapping its shared memory regions and perform NVMe operations including creating queue pairs.

This feature is enabled by default and is controlled by selecting a value for the shared memory group ID. This ID is a positive integer and two applications with the same shared memory group ID will share memory. The first application with a given shared memory group ID will be considered the primary and all others secondary.

Example: identical shm_id and non-overlapping core masks

./perf options [AIO device(s)]...
	[-c core mask for I/O submission/completion]
	[-i shared memory group ID]

./perf -q 1 -s 4096 -w randread -c 0x1 -t 60 -i 1
./perf -q 8 -s 131072 -w write -c 0x10 -t 60 -i 1

Scalability and Performance

To maximize the I/O bandwidth of an NVMe device, ensure that each application has its own queue pairs.

The optimal threading model for SPDK is one thread per core, regardless of which processes that thread belongs to in the case of multi-process environment. To achieve maximum performance, each thread should also have its own I/O queue pair. Applications that share memory should be given core masks that do not overlap.

However, admin commands may have some performance impact as there is only one admin queue pair per NVMe SSD. The NVMe driver will automatically take a cross-process capable lock to enable the sharing of admin queue pair. Further, when each process polls the admin queue for completions, it will only see completions for commands that it originated.

Limitations

  1. Two processes sharing memory may not share any cores in their core mask.
  2. If a primary process exits while secondary processes are still running, those processes will continue to run. However, a new primary process cannot be created.
  3. Applications are responsible for coordinating access to logical blocks.

@sa spdk_nvme_probe, spdk_nvme_ctrlr_process_admin_completions