e8ea27f859
This suite can be used to deploy containers with the following functionality (more details in README.md): - storage-target - proxy-container - traffic-generator This will run simple fio test as per fio.conf against nvmf controller provided by initiator-container. Similar task can be performed directly from initiator-container as well. Each container includes SPDK installation with most common tools, e.g. rpc.py, available under $PATH. This allows for something like: docker-compose exec storage-target rpc.py nvmf_get_subsystems Note that SPDK environment heavily depends on a running kernel hence all the containers need to be privileged. That said, to make sure containers are not affecting the host too much, some tasks must be done prior running them. This includes: - loading proper kernel modules (like nvme-fabrics, etc.) - allocating hugepages and having at least one hugetlbfs mount available under /dev/hugepages base_build is created as docker multi-stage build. This is done in order to decrease the size of the final image. The SPDK RPMs are built inside a base image and then copied over to the main image (+ fio binary) - this leaves all the dependencies inside the intermediate image instead of the final one. The resulted difference in size may look similar to the following (it may differ depending on the docker version etc.): no multi-stage build: spdk_base == 1.04GB multi-stage build: spdk_base == 261MB Signed-off-by: Michal Berger <michalx.berger@intel.com> Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com> Change-Id: I825bd0d0bb4071bd9d44b6a0749c033894899ae0 Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9055 Reviewed-by: Jim Harris <james.r.harris@intel.com> Reviewed-by: Monica Kenguva <monica.kenguva@intel.com> Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com> Reviewed-by: Changpeng Liu <changpeng.liu@intel.com> Tested-by: SPDK CI Jenkins <sys_sgci@intel.com> Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
SPDK Docker suite
This suite is meant to serve as an example of how SPDK can be encapsulated into docker container images. The example containers consist of SPDK NVMe-oF target sharing devices to another SPDK NVMe-oF application. Which serves as both initiator and target. Finally a traffic generator based on FIO issues I/O to the connected devices.
Prerequisites
docker: We recommend version 20.10 and above because it supports cgroups v2 for customization of host resources like CPUs, memory, and block I/O.
docker-compose: We recommend using 1.29.2 version or newer.
kernel: Hugepages must be allocated prior running the containers and hugetlbfs mount must be available under /dev/hugepages. Also, tmpfs should be mounted under /dev/shm. Depending on the use-case, some kernel modules should be also loaded into the kernel prior running the containers.
proxy: If you are working behind firewall make sure dockerd is aware of the proxy. Please refer to: docker-proxy
To pass $http_proxy to docker-compose build use:
docker-compose build --build-arg PROXY=$http_proxy
How-To
docker-compose.yaml shows an example deployment of the storage containers based on SPDK.
Running docker-compose build creates 4 docker images:
- build_base
- storage-target
- proxy-container
- traffic-generator
The build_base image provides the core components required to containerize SPDK
applications. The fedora:33 image from the Fedora Container Registry is used and then SPDK is installed. SPDK is installed out of build_base/spdk.tar.gz provided.
See build_base folder for details on what's included in the final image.
Running docker-compose up creates 3 docker containers:
-- storage-target: Contains SPDK NVMe-oF target exposing single subsystem to
proxy-container based on malloc bdev.
-- proxy-container: Contains SPDK NVMe-oF target connecting to storage-target
and then exposing the same subsystem to traffic-generator.
-- traffic-generator: Contains FIO using SPDK plugin to connect to proxy-container
and runs a sample workload.
Each container is connected to a separate "spdk" network which is created before
deploying the containers. See docker-compose.yaml for the network's detailed setup and ip assignment.
All the above boils down to:
cd docker
tar -czf build_base/spdk.tar.gz --exclude='docker/*' -C .. .
docker-compose build
docker-compose up
The storage-target and proxy-container can be started as services.
Allowing for multiple traffic-generator containers to connect.
docker-compose up -d proxy-container
docker-compose run traffic-generator
Enviroment variables to containers can be passed as shown in docs. For example extra arguments to fio can be passed as so:
docker-compose run -e FIO_ARGS="--minimal" traffic-generator
As each container includes SPDK installation it is possible to use rpc.py to examine the final setup. E.g.:
docker-compose exec storage-target rpc.py bdev_get_bdevs
docker-compose exec proxy-container rpc.py nvmf_get_subsystems
Caveats
- If you run docker < 20.10 under distro which switched fully to cgroups2 (e.g. f33) make sure that /sys/fs/cgroup/systemd exists otherwise docker/build will simply fail.
- Each SPDK app inside the containers is limited to single, separate CPU.