wawryk 1e1fd9ac21 markdownlint: enable rule MD025

MD025 - Multiple top level headers in the same document
Fixed all errors
Update check_format.sh to fit new header style in jsonrpc.md

Signed-off-by: Maciej Wawryk <maciejx.wawryk@intel.com>
Change-Id: Ib5f832c549880771c99c15b89affe1e82acd3fa4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9045
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

2021-08-26 19:27:22 +00:00

2.7 KiB

Raw Blame History

Peer-2-Peer DMAs

Please note that the functionality discussed in this document is currently tagged as experimental.

In this document

@ref p2p_overview
@ref p2p_nvme_api
@ref p2p_cmb_copy
@ref p2p_issues

Overview

Peer-2-Peer (P2P) is the concept of DMAing data directly from one PCI End Point (EP) to another without using a system memory buffer. The most obvious example of this from an SPDK perspective is using a NVMe Controller Memory Buffer (CMB) to enable direct copies of data between two NVMe SSDs.

In this section of documentation we outline how to perform P2P operations in SPDK and outline some of the issues that can occur when performing P2P operations.

The P2P API for NVMe

The functions that provide access to the NVMe CMBs for P2P capabilities are given in the table below.

Key Functions	Description
spdk_nvme_ctrlr_map_cmb()	@copybrief spdk_nvme_ctrlr_map_cmb()
spdk_nvme_ctrlr_unmap_cmb()	@copybrief spdk_nvme_ctrlr_unmap_cmb()
spdk_nvme_ctrlr_get_regs_cmbsz()	@copybrief spdk_nvme_ctrlr_get_regs_cmbsz()

Determining device support

SPDK's identify example application displays whether a device has a controller memory buffer and which operations it supports. Run it as follows:

./build/examples/identify -r traddr:<pci id of ssd>

cmb_copy: An example P2P Application

Run the cmb_copy example application.

./build/examples/cmb_copy -r <pci id of write ssd>-1-0-1 -w <pci id of write ssd>-1-0-1 -c <pci id of the ssd with cmb>

This should copy a single LBA (LBA 0) from namespace 1 on the read NVMe SSD to LBA 0 on namespace 1 on the write SSD using the CMB as the DMA buffer.

Issues with P2P

In some systems when performing peer-2-peer DMAs between PCIe EPs that are directly connected to the Root Complex (RC) the DMA may fail or the performance may not be great. Basically your milage may vary. It is recommended that you use a PCIe switch (such as those provided by Broadcom or Microsemi) as that is know to provide good performance.
Even with a PCIe switch there may be occasions where peer-2-peer DMAs fail to work. This is probably due to PCIe Access Control Services (ACS) being enabled by the BIOS and/or OS. You can disable ACS using setpci or via out of tree kernel patches that can be found on the internet.
In more complex topologies involving several switches it may be possible to construct multiple paths between EPs. This could lead to TLP ordering problems. If you are working in these environments be careful!

2.7 KiB Raw Blame History