diff --git a/test/config/README.md b/test/config/README.md new file mode 100644 index 0000000000..b486521043 --- /dev/null +++ b/test/config/README.md @@ -0,0 +1,79 @@ +# Virtual Test Configuration + +This readme and the associated bash script, vm_setup.sh, are intended to assist developers in quickly +preparing a virtual test environment on which to run the SPDK validation tests rooted at autorun.sh. +This file contains basic information about SPDK environment requirements, an introduction to the +autorun-spdk.conf files used to moderate which tests are run by autorun.sh, and step-by-step instructions +for spinning up a VM capable of running the SPDK test suite. +There is no need for external hardware to run these tests. The linux kernel comes with the drivers necessary +to emulate an RDMA enabled NIC. NVMe controllers can also be virtualized in emulators such as QEMU. + + +## VM Envronment Requirements (Host): +- 8 GiB of RAM (for DPDK) +- Enable intel_kvm on the host machine from the bios. +- Enable nesting for VMs in kernel command line (for vhost tests). + - In `/etc/default/grub` append the following to the GRUB_CMDLINE_LINUX line: intel_iommu=on kvm-intel.nested=1. + +## VM Specs +When creating the user during the fedora installation, it is best to use the name sys_sgsw. Efforts are being made +to remove all references to this user, or files specific to this user from the codebase, but there are still some +trailing references to it. + +## Autorun-spdk.conf +Every machine that runs the autotest scripts should include a file titled autorun-spdk.conf in the home directory +of the user that will run them. This file consists of several lines of the form 'variable_name=0/1'. autorun.sh sources +this file each time it is run, and determines which tests to attempt based on which variables are defined in the +configuration file. For a full list of the variable declarations available for autorun-spdk.conf, please see +`scripts/autotest_common.sh` starting at line 13. + +## Steps for Configuring the VM +1. Download a fresh Fedora 26 image. +2. Perform the installation of Fedora 26 server. +3. Create an admin user sys_sgsw (enabling passwordless sudo for this account will make life easier during the tests). +4. Run the vm_setup.sh script which will install all proper dependencies. +5. Modify the autorun-spdk.conf file in the home directory. +6. Reboot the VM. +7. Run autorun.sh for SPDK. Any output files will be placed in `~/spdk_repo/output/`. + +## Additional Steps for Preparing the Vhost Tests +The Vhost tests also require the creation of a second virtual machine nested inside of the test VM. +Please follow the directions below to complete that installation. Note that host refers to the Fedora VM +created above and guest or VM refer to the Ubuntu VM created in this section. +1. Create an image file for the VM. It does not have to be large, about 3.5G should suffice. +2. Create an ssh keypair for host-guest communications (performed on the host): + - Generate an ssh keypair with the name spdk_vhost_id_rsa and save it in `/root/.ssh`. + - Make sure that only root has read access to the private key. +3. Install the OS in the VM image (performed on guest): + - Use the latest Ubuntu server (Currently 16.04 LTS). + - When partitioning the disk, make one partion that consumes the whole disk mounted at /. Do not encrypt the disk or enable LVM. + - Choose the OpenSSH server packages during install. +4. Post installation configuration (performed on guest): + - Run the following commands to enable all necessary dependencies: + ~~~{.sh} + sudo apt update + sudo apt upgrade + sudo apt install fio sg3-utils bc + ~~~ + - Enable the root user: "sudo passwd root -> root". + - Enable root login over ssh: vim `/etc/ssh/sshd_config` -> PermitRootLogin=yes. + - Disable DNS for ssh: `/etc/ssh/sshd_config` -> UseDNS=no. + - Add the spdk_vhost key to root's known hosts: `/root/.ssh/authorized_keys` -> add spdk_vhost_id_rsa.pub key to authorized keys. + Remember to save the private key in `~/.ssh/spdk_vhost_id_rsa` on the host. + - Change the grub boot options for the guest as follows: + - Add "console=ttyS0 earlyprintk=ttyS0" to the boot options in `/etc/default/grub` (for serial output redirect). + - Add "scsi_mod.use_blk_mq=1" to boot options in `/etc/default/grub`. + ~~~{.sh} + sudo update-grub + ~~~ + - Reboot the VM. + - Remove any unnecessary packages (this is to make booting the VM faster): + ~~~{.sh} + apt purge snapd + apt purge Ubuntu-core-launcher + apt purge squashfs-tools + apt purge unattended-upgrades + ~~~ +5. Copy the fio binary from the guest location `/usr/bin/fio` to the host location `/home/sys_sgsw/fio_ubuntu`. +6. Place the guest VM in the host at the following location: `/home/sys_sgsw/vhost_vm_image.qcow2`. +7. On the host, edit the `~/autorun-spdk.conf` file to include the following line: SPDK_TEST_VHOST=1. diff --git a/test/config/vm_setup.sh b/test/config/vm_setup.sh new file mode 100644 index 0000000000..5a7c3cb0aa --- /dev/null +++ b/test/config/vm_setup.sh @@ -0,0 +1,168 @@ +#! /usr/bin/env bash + +# Virtual Machine environment requirements: +# 8 GiB of RAM (for DPDK) +# enable intel_kvm on your host machine + +# The purpose of this script is to provide a simple procedure for spinning up a new +# virtual test environment capable of running our whole test suite. This script, when +# applied to a fresh install of fedora 26 server will install all of the necessary dependencies +# to run almost the complete test suite. The main exception being VHost. Vhost requires the +# configuration of a second virtual machine. instructions for how to configure +# that vm are included in the file TEST_ENV_SETUP_README inside this repository + +# it is important to enable nesting for vms in kernel command line of your machine for the vhost tests. +# in /etc/default/grub +# append the following to the GRUB_CMDLINE_LINUX line +# intel_iommu=on kvm-intel.nested=1 + +# We have made a lot of progress with removing hardcoded paths from the tests, +# but it may be easiest if you create your user with the name sys_sgsw for now. + +set -e + +sudo dnf upgrade -y +sudo dnf install -y git +sudo dnf install -y jq +sudo dnf install -y valgrind +sudo dnf install -y nvme-cli +sudo dnf install -y ceph +sudo dnf install -y gdb +sudo dnf install -y sg3_utils +sudo dnf install -y fio +sudo dnf install -y librbd-devel +sudo dnf install -y kernel-devel +sudo dnf install -y gflags-devel +sudo dnf install -y libubsan +sudo dnf install -y autoconf +sudo dnf install -y automake +sudo dnf install -y libtool +sudo dnf install -y libmount-devel +sudo dnf install -y isns-utils-devel +sudo dnf install -y openssl-devel +sudo dnf install -y numactl-devel +sudo dnf install -y libaio-devel +sudo dnf install -y CUnit-devel +sudo dnf install -y clang-analyzer +sudo dnf install -y libpmemblk-devel +sudo dnf install -y libibverbs libibverbs-devel librdmacm librdmacm-devel +sudo dnf install -y perl-open +sudo dnf install -y glib2-devel +sudo dnf install -y pixman-devel +sudo dnf install -y libiscsi-devel +sudo dnf install -y doxygen + +cd ~ + +mkdir spdk_repo + +# the configurations of username and email are needed later for applying patches to iscsiadm. +git config --global user.name none +git config --global user.email none + +# The librxe-dev repository provides a command line tool called rxe_cfg which makes it +# very easy to use Soft-RoCE. The build pool utilizes this command line tool in the absence +# of any real RDMA NICs to simulate one for the NVMe-oF tests. +git clone https://github.com/SoftRoCE/librxe-dev.git +sudo dnf install -y perl-Switch librdmacm-utils libibverbs-utils + +cd librxe-dev +./configure --libdir=/usr/lib64/ --prefix= +make +sudo make install +cd ~ + +cd spdk_repo +mkdir output +git clone https://review.gerrithub.io/spdk/spdk +cd spdk +git submodule update --init --recursive +cd ~ + +# The version of iscsiadm that ships with fedora 26 was broken as of November 3 2017. +# There is already a bug report out about it, and hopefully it is fixed soon, but in the event that +# that version is still broken when you do your setup, the below steps will fix the issue. +CURRENT_VERSION=$(iscsiadm --version) +OPEN_ISCSI_VER='iscsiadm version 6.2.0.874' +if [ "$CURRENT_VERSION" == "$OPEN_ISCSI_VER" ]; then + mkdir open-iscsi-install + cd open-iscsi-install + sudo dnf download --source iscsi-initiator-utils + rpm2cpio iscsi-initiator-utils-6.2.0.874-3.git86e8892.fc26.src.rpm | cpio -idmv + mkdir patches + mv 00* patches/ + git clone https://github.com/open-iscsi/open-iscsi + + cd open-iscsi + git checkout 86e8892 + for patch in `ls ../patches`; do + git am ../patches/$patch + done + sed -i '427s/.*/-1);/' usr/session_info.c + make + sudo make install + cd ~ +fi + + +sudo mkdir -p /usr/src + +# Rocksdb is installed for use with the blobfs tests. +git clone https://review.gerrithub.io/spdk/rocksdb +git -C ./rocksdb checkout spdk-v5.6.1 +sudo mv rocksdb /usr/src/ + +git clone http://git.kernel.dk/fio.git +sudo mv fio /usr/src/ +( + cd /usr/src/fio && + make && + sudo make install +) +cd ~ + +git clone https://github.com/brendangregg/FlameGraph.git +mkdir -p /usr/local +sudo mv FlameGraph /usr/local/FlameGraph + + +# The SPDK branch of Qemu contains some extra functionality needed by SPDK. However, that branch is not currently +# compatible with GCC 7. Some of the SPDK changes are currently out for review on Qemu's main branch. Until those +# changes are merged, this specific review provides support for both spdk and GCC 7. +mkdir vhost +cd vhost +git clone https://review.gerrithub.io/spdk/qemu +cd qemu +./configure --prefix=/home/sys_sgsw/spdk_repo/root --target-list="x86_64-softmmu" --enable-kvm --enable-linux-aio --enable-numa +sudo make install +cd ~ + +# create autorun-spdk.conf in home folder. This is sourced by the autotest_common.sh file. +# By setting any one of the values below to 0, you can skip that specific test. If you are +# using your autotest platform to do sanity checks before uploading to the build pool, it is +# probably best to only run the tests that you believe your changes have modified along with +# Scanbuild and check format. This is because running the whole suite of tests in series can +# take ~40 minutes to complete. +cat > ~/autorun-spdk.conf << EOF +# assign a value of 1 to all of the pertinent tests +SPDK_BUILD_DOC=1 +SPDK_BUILD_IOAT_KMOD=1 +SPDK_RUN_CHECK_FORMAT=1 +SPDK_RUN_SCANBUILD=1 +SPDK_RUN_VALGRIND=1 +SPDK_TEST_UNITTEST=1 +SPDK_TEST_ISCSI=1 +SPDK_TEST_NVME=1 +SPDK_TEST_NVMF=1 +SPDK_TEST_RBD=1 +# requires some extra configuration. see TEST_ENV_SETUP_README +SPDK_TEST_VHOST=0 +SPDK_TEST_BLOCKDEV=1 +# doesn't work on vm +SPDK_TEST_IOAT=0 +SPDK_TEST_EVENT=1 +SPDK_TEST_BLOBFS=1 +SPDK_TEST_NVML=1 +SPDK_RUN_ASAN=1 +SPDK_RUN_UBSAN=1 +EOF