We used to crash when `mount` returned
multiple hugetlb mountpoints.
Change-Id: I75717c2c394204bdfec730edabb9ff6386047742
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/395926
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Patch da7e3bb8 [1] disabled chown for vfio
groups and hugepage mountpoints. Chown code
was never being executed.
[1] da7e3bb8 ("setup.sh: replace username param with TARGET_USER env
variable")
Change-Id: Ie5c9ac42fca5b60c72ced501d4065842a5bef9cd
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/395925
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Make it configurable the same way as other options.
setup.sh will configure hugepages and vfio for the
current user by default. To specify other user,
set TARGET_USER variable.
Change-Id: Ib98178a70a007bec761f7465689c2150e8844ffb
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/394806
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
In some cases we may not want to assign all PCIe NVMe controllers in a
system to SPDK. Add a new input to the setup.sh script called
NVME_WHITELIST which whitelists (via PCIe slot ID) the NVMe
controllers you wish to add to SPDK.
If the NVME_WHITELIST input argument is not defined then all PCIe NVMe
controllers will be added. The values in the whitelist whould be
white-space seperated and the entire list should be enclosed in double
quotes ("").
To blacklist all PCIe NVMe devices use a non-valid PCIe slot ID
(e.g. the string "none" would work very well).
Examples:
NVME_WHITELIST="0000:02:00.0" ./setup.sh
NVME_WHITELIST="0000:08:00.0 0000:06:00.1" ./setup.sh
NVME_WHITELIST="none" ./setup.sh
Change-Id: If6ebb04307180cbac11fc41cd9a70749640bc598
Signed-off-by: Stephen Bates <sbates@raithlin.com>
Reviewed-on: https://review.gerrithub.io/394303
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Very old kernels (e.g. 3.13 from Ubuntu 14.04) don't have the nvme
directory in /sys/bus/pci/devices/$bdf/, so check for its existence
before trying to list it.
Change-Id: I1f1ca04c71de6359de2b924fb9437ad03c56523b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/393725
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This makes us iterate through all devices
on a controller and check if any of those
is mounted. Previously we only checked the
very first device, which is obviously
insufficient.
Change-Id: I776846b931480d1e0a059137950c21f057ecb710
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/393749
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Add a new file, scripts/common.sh, that can be shared between not only
autotest scripts, but also scripts/setup.sh, scripts/gen_nvme.sh, etc.,
and move the PCI iteration functions there.
The iterators are also expanded to work identically for both dev_id and
class_code on Linux and FreeBSD.
Change-Id: I98423cd06242e78535f5da4fce82166812ea96a8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/393416
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
When looking up nvme devices by name in older linux kernel versions, setup.sh
was throwing errors while trying to follow a symlink on
/sys/block/. This patch adds a fallback to the proper location of this
symlink in those versions of the kernel.
Change-Id: I13b5c1ce6f7505251789b64c740aba04d67f39a9
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/391019
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
User may only want to configure hugepage and no configure
on the PCI devices. Add the SKIP_PCI parameter to offer
this option.
If user only wants to configure hugepage, can run:
SKIP_PCI=1 ./scripts/setup.sh
The default behavior is that PCI devices will be configured.
Change-Id: I8302efd5a09f2885e00fa8ed00d398219c11dde7
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/389473
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Introduce several new functions configure_linux_pci, reset_linux_pci and
configure_freebsd_pci for the PCI devices relatd operations.
Change-Id: Ifdaddd314ba01fd73ff700d231a6aeae5ec7c480
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/387265
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The setup.sh can now be invoked as following:
```
HUGENODE=0 NRHUGE=5 ./scripts/setup.sh
HUGENODE=1 NRHUGE=3 ./scripts/setup.sh
```
This will allocate 5 hugepages on node0,
and 3 hugepages on node1. If HUGENODE
param is not specified, the setup.sh will
operate just like it used to.
Change-Id: Ib198f5f32203abcc646af3c8d823f2b3e9bce362
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/384086
Reviewed-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
The previous defines have been
removed by patch 80b44ea.
Fixes: 80b44ea ("rte_virtio: remove unused PCI ID #defines")
Change-Id: I0e90e7f91a5e91efca3e5562b94cb1f6a36e7428
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/382935
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use lsblk to detect if an NVMe namespace or any partition on the
namespace has an active mountpoint. If it does, do not unbind
the NVMe device associated with that NVMe namespace.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I1ab7540d640baa201efac49bc9515fd861dd8f8c
Reviewed-on: https://review.gerrithub.io/382479
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
If the user did not have the ioatdma driver loaded for example,
when running "setup.sh reset", we should not load ioatdma so that
we can bind to it. Instead just unbind the devices from the
uio/vfio driver.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I953941eb4918105b97ad78987b47b33f4372ae01
Reviewed-on: https://review.gerrithub.io/382474
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This allows to use same command
for allocating hugepages on
systems with different hugepage
sizes.
The old NRHUGE variable can be
still set on Linux machines,
but is currently undocumented.
Change-Id: I1fba315d95e9aae8b6a8c9c445deb447fecc65dc
Signed-off-by: Hailiang Wang <hailiangx.e.wang@intel.com>
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/369546
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
When a DPDK app closes, it leaves behing configuration files that can
cause latent failures in the build pool machines if they are not
removed.
Change-Id: I80caa1b595e6586a109c013b92dfcf306d7a0150
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/379874
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Supports both PCI mode (for usage in guest VMs) and
vhost-user mode (for usage in host processes). The rte_virtio
subdirectory contains a lot of code lifted from the DPDK
virtio-net driver. Most of the PCI and vhost-user code is
reused almost exactly as-is, but the virtio code is drastically
rewritten as the DPDK code was very network specific.
Has been lightly tested with both the bdevio and bdevperf
applications in both PCI and vhost-user modes.
Still quite a bit of work needed - a list of todo
items is included in a README in the module's directory.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I85989d3de9ea89a87b719ececdb6d2ac16b77f53
Reviewed-on: https://review.gerrithub.io/374519
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Another atempt to fix setup.sh On Ubuntu 16.04 that use GNU Awk 4.1.3.
Eg when subsystem vendor name and subsystem name are empty:
$ lspci -mm -n -D
...
0000:00:05.0 "0880" "8086" "0e28" -r04 "8086" "3582"
0000:00:05.1 "0880" "8086" "0e29" -r04 "" ""
0000:00:05.2 "0880" "8086" "0e2a" -r04 "8086" "3582"
...
tr -d '"' part is brekaing whole awk script. So lets try to escape
vendor and device ID to comapre them as whole string.
Change-Id: Ic41e82c41347c6e0a8246b445f86eddf56780089
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Reviewed-on: https://review.gerrithub.io/374462
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Device IDs starting with 0e get treated as 0 due to numerical
interpretation (0 * 10^x = 0). So use ~ to do a string regexp
comparison instead.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4b5558c0127b0c4f021daf8151bf4d3f514e44da
Reviewed-on: https://review.gerrithub.io/374507
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
1) Create separate functions for checking the PCI class code
vs. checking the PCI device/vendor IDs
2) Instead of a simple grep, check the specific fields to
ensure we avoid accidental matching of a class code
against a device ID (or vice versa)
This will make it a bit cleaner to add some new device
IDs (i.e. virtio devices) in some future commits.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I36d4e2dec2f0f15dc0d603fc2512ea03a64dd560
Reviewed-on: https://review.gerrithub.io/374014
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
On some systems, hugetlbfs is reported as being mounted on 'nodev'
rather than 'hugetlbfs'; use the type instead of the device to match the
hugetlbfs filesystem.
Change-Id: I56498b7cf01de9759669fe313048b43b2319b933
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/368228
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Adding group write permission is needed in case of using vfio
driver and using libvirtd for vm provisioning to avoid
permission errors.
Change-Id: I5348e74c65bd8639071c1a2a164f28849dde9678
Signed-off-by: Karol Latecki <karolx.latecki@intel.com>
If we created /mnt/huge due to no hugetlbfs mount point existing, we
should also use that path in the chown step for VFIO.
Change-Id: I11614a67b16dbda6626b9adf58d8faea10205cab
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This scripts adds new parameter "status", which shows extra
information about devices connected to the system:
- Numa node on which device is located
- Currently bound driver
- Device name in OS if available
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: If77d5fc20330b2492afd87bf832664c450705624
Default is still 1024 2MB huge pages, but user can
override this by setting a different value in the
environment.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8d189d3a71f474469a44b6eb41b4443591863f3c
In some circumstances, no utmp entry is available, so logname fails.
In particular, gnome-terminal no longer creates a utmp entry:
https://bugzilla.gnome.org/show_bug.cgi?id=747046
As a workaround, try $SUDO_USER - the use case here is to determine
the (unprivileged) user name so we can give them ownership of
certain files, so this is usually the right thing to use anyway.
If we are not running under sudo, the caller should have passed the
username as a parameter to scripts/setup.sh anyway, since we can't
reliably determine which user is intended.
Also check if username is actually set before using it to run chmod - it
is possible that the scripts/setup.sh caller does not want to provide
access to an unprivilieged user and just wants to run everything as
root.
Change-Id: I20631c325b52884a378029dcf38568a2b311b457
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Fix typo in checking for mounted hugepages and replace expr with
arithmetic context as expr is returning error when expression is zero.
Change-Id: I2433fa938b3b2e758e1a4267414c10c5a3a5caff
Signed-off-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
This patch also drops support for automatically unbinding
devices from the kernel - run scripts/setup.sh first.
Our generic pci interface is now hidden behind include/spdk/pci.h
and implemented in lib/util/pci.c. We no longer wrap the calls
in nvme_impl.h or ioat_impl.h. The implementation now only uses
DPDK and the libpciaccess dependency has been removed. If using
a version of DPDK earlier than 16.07, enumerating devices
by class code isn't available and only Intel SSDs will be
discovered. DPDK 16.07 adds enumeration by class code and all
NVMe devices will be correctly discovered.
Change-Id: I0e8bac36b5ca57df604a2b310c47342c67dc9f3c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Allocate larger contiguous regions (but the same amount of total
memory).
Change-Id: I54d09c11afdc7bdf232cc3b0864719c063a74b1a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This increases the largest contiguous region available for any single
object (e.g. DPDK mempools) from 32 MB to 64 MB.
Change-Id: I46633b246bcf7843053ab9d0d92993a784287a57
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For now, drop the option to configure the number of hugepages (just use
the default 1024).
Change-Id: I686b6151e4a32ffa4ae9a9cf2694693369e03f49
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
cleanup.sh and unbind.sh have been combined into a single
setup.sh that takes one optional parameter (reset). If no
parameter is given, the script will automatically bind
all NVMe and IOAT devices to either uio_pci_generic
or vfio-pci, as appropriate based on IOMMU settings. If
the reset parameter is given, the devices will be bound back
to the appropriate kernel drivers.
Change-Id: I25db3234f1ecfb352a281e5093f4c1aa455152ae
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
