For the purpose to make performance comparison between the ioat
kernel driver and user space driver, we added the kernel driver
test harness here, all the workload executed in the kernel space
and controlled via sysfs.
Change-Id: I2c8d826283405a5e1c9ba6a033503bcb98541370
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Use /sys/bus/pci/devices/.../driver to determine which driver is loaded
for a particular device.
Change-Id: I5859a776e524033e1c6d6ec3796b7e11bdcf0bc4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This more accurately represents what function it performs.
Also remove pci_device_has_uio_driver() from the public API. Callers
should use pci_device_has_non_uio_driver() instead.
Change-Id: I9623fe1345b43e981d5823804e33d01ac0d3bb1c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_ctrlr_process_io_completions() and
nvme_ctrlr_process_admin_completions() now return the number of
completions processed.
This also adds the possibility of returning an error from the
process_*_completions functions (currently unused, but this at least
gets the API ready in case error conditions are added later).
Change-Id: I1b32ee4f2f3c1c474d646fa2d6b8b7bbb769785f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This doesn't really matter, since the program will be exiting
immediately if associate_workers_with_ns() fails, but it makes static
analyzers happy.
Change-Id: Ic21d234dec50bd2b6684b5fe2caa78d616f93052
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
_Static_assert is new in C11, and older versions of GCC don't support
it. Detect whether static_assert is available in <assert.h> and if not,
define a fallback macro.
Change-Id: Iaf513677e5f356832e0a3d1f6b1f4f9ea07e6627
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make sure allocation succeed in the unit tests.
Change-Id: I1c9dd64c256d529f2e40be49448d56359969a6f5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The normal CU_ASSERT_FATAL macro calls a function that is not marked as
noreturn, so static analyzers (e.g. scan-buid) can't figure out that
fatal asserts are really fatal.
Add a wrapper macro that calls abort(), which the analyzer can determine
does not return.
Change-Id: I0c087bf9c8d3c272bf88120caa70e87dab6d9546
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This ensures that any uninitialized fields are 0/NULL so if
ioat_channel_start() fails, ioat_channel_destruct() will not try to free
bogus pointers.
Change-Id: I99278c9fa280cbcdf3f7448e77db3ac98b59cdd6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Previously, if nvme_allocate_request() failed in
nvme_ctrlr_construct_and_submit_aer(), there was no error checking, so a
NULL pointer would be dereferenced.
Add a return value to nvme_ctrlr_construct_and_submit_aer() so we can
signal failure to the caller. This can only really be reasonably
handled during initialization; when resubmitting a completed AER later,
there is nowhere to report failure, so the AER will just remain
unsubmitted.
Change-Id: I413eb6c21be01cd9a61e67f62f2d0b7170eabaa3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
If any command fails during the asynchronous event request test, exit
the test with an error rather than waiting for an event that won't
arrive.
Change-Id: I7297afe362bfdac04dd6585cc97ffdceb9f0096e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The top-level autotest.sh script will catch any core dumps at the end of
the test run, so sprinkling process_core in the individual test scripts
is unnecessary.
Also make the per-component test scripts run with 'set -e' (exit on
error).
Change-Id: I85f124e164ca93d35eaf672a428a841c119c550b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Switch to /usr/local/share/dpdk on Linux to use the latest installed
DPDK.
This is the filesystem layout from building DPDK with:
make install T=x86_64-native-linuxapp-gcc DESTDIR=/usr/local
Change-Id: I95cb0a3ee2ad1fcc89f6abb73ff0005aa65110e5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
DPDK's install target now requires DESTDIR to be set, so change the
build instructions to set DESTDIR=. to match previous behavior.
Change-Id: Ib697c2f54704210a5b60278ba1a5b20a16f517be
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
It was just using 16 make threads by default before.
Now it uses the number of CPU cores.
Change-Id: I695705eb73fc5f5ed3eafe1f85c686bde0e4b7d6
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
The deallocate command's num_ranges have changed to unit16_t.
Update the nvme_ns_cmd_deallocate unit test for the change.
Change-Id: I43b8637bbb953b0e56c39998e1e6682a54304a8f
Signed-off-by: Liang Yan <liangx.yan@intel.com>
The ioat library currently only supports DMA copy operations, but the
hardware can do other types of transfers. Add a union of the hardware
descriptor structures to enable support for the other operations in the
future.
Also add a generic hardware descriptor type to allow access to the parts
of the descriptor that are common between all types.
Change-Id: I3b54421ce771f58b78910e790b53026f311f918e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
If the I/O size is larger than the total namespace size or smaller than
the block size, ignore that namespace in the perf utility.
Change-Id: I297303d8c41ceb36eef91c6c33da809a35758f4e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows identify to print the raw data so that the user can look up
fields that aren't yet formatted in a human-readable way.
Change-Id: If3db1fed85fee06c8b8c18246a2ea253b1e6ae13
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This patch is used to remove the unused pci_dev parameter from the nvme
perf utility functions that no longer need it.
Change-Id: Ib139b080b7668aed712b4489c5ee95bd2fa2b350
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
According to the specification, the dataset management for deallocate
attribute can support to 256 ranges, so we should use uint16_t
instead of uint8_t as the ranges parameter.
Change-Id: Ibacc00da8b4b9e2b2f3454d382aadf7ad353ff31
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The parent field is no longer used in the normal (non-split) I/O path,
so move it down to the default-uninitialized part of struct nvme_request
that is only touched for parent/child I/O.
This also puts it closer to other related fields (children,
child_tailq, parent_status) for improved readability.
Change-Id: I120df1df0c967d2f74daa6e97c0bc83626e3be7f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Add support for the full set of Optional NVM Command Support bits from
NVMe 1.2, and interpret the various SGL-related feature bits.
Change-Id: Ie3fcca27571b5236479e05a1ea3b834adbeb18a8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_qpair_submit_tracker() and nvme_qpair_manual_complete_request() are
only used from within nvme_qpair.c, so they can be static.
nvme_qpair_submit_tracker() is moved up to avoid needing a declaration
(no other code change).
nvme_ctrlr_hw_reset() is only used from within nvme_ctrlr.c, so it can
be static.
Change-Id: I9a7953d7baaec76e875dd535daf557ea24bef801
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
These delays are left over from early development. They are completely
unnecessary and not based on anything in the NVMe spec.
Startup time should be slightly improved (on the order of 100 ms in
normal cases).
Change-Id: I9068b1a0f42feabcfe656d68be91e05a56cc53a3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Some compilers automatically define _FORTIFY_SOURCE to a different
value, so undefine it before setting it on the compiler command line.
Change-Id: Id662953bd4466b99d0fe9bd5bb96f9fc335ff9a8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This lets us signal an error if the channel is halted in
ioat_process_channel_events().
Change-Id: Iffaf4fd1e27d1254f9d95a37d732ae4a5f3a0465
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rather than individually allocating each ring entry, use two large
allocations, one for the hardware descriptors and one for the software
descriptor contexts.
This allows the use of simple array indexing on the rings and also
allows the removal of most of the software descriptor structure,
since the necessary information can be retrieved based on the ring
index now.
Change-Id: I73ef24450f69ca0fc35e350286282c6b1c77a207
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
NVMe doesn't require the specific 64-bit MMIO ordering on 32-bit
platforms performed in spdk_mmio_read_8(), but it doesn't hurt.
We have to pick one of the two possible orderings, so pick the one
required by I/OAT.
Change-Id: I2b909d64d0c077b797d0f64a11d78d1ecc55eec7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Previously, the cleanup functions were only being called if there was an
error during initialization.
Change-Id: I1606cfa9a9c3732d670131f78249d34a5db47403
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The ioat driver supports DMA engine copy offload hardware available on
Intel Xeon platforms.
Change-Id: Ida0b17b25816576948ddb1b0443587e0f09574d4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Make the CONFIG_COVERAGE definition in CONFIG consistent with the other
ones using ?=.
Change-Id: I5994b936eebf80dfff9733d38c842c4bb214c591
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Normal users building SPDK probably won't read CONFIG, so provide a
reasonable default of release-mode builds.
Change-Id: I2609aa445eabf83bd2cdbe79c548d8c0da7db7e1
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Also request fewer huge pages overall during autotest.
Change-Id: I6744c9a948e8db24b8ed3c7d9d99c89e1e50ba9c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Use $(CLEAN_C) throughout the Makefiles to clean up a consistent set of
generated files.
This also adds coverage files to the list of cleaned files.
Change-Id: Iceb922935a45c9eecbf2f3443bd0ee4f5c966825
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is very convenient. It also works from subdirectories, as
long as you adjust the relative path you pass appropriately.
Change-Id: I8cfeac380ccb5fad4389ee1feb838f986a837f11
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Sometimes readelf prepends an "fname: " note, to the line
with "psargs", and in that case just using awk with $2 does
not work.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6695352da641111f90be9f3e519accbf05107e91
Most devices today support far fewer than 1024, but this is a
more reasonable default upper limit than the spec-defined 64K.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ia8a6d80c3a5aa181f27c8354758c6ca468013d92
librte_malloc has been deprecated by DPDK, but to allow for backwards
compability, check for its existence and link with it if it exists.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I3e05a3295b6768bd6804a8ec86c301e22040df37
lib/memory was already using this pattern; extend it to lib/util and
lib/nvme.
Change-Id: I84a633d7805522fc94d8fc11ad5486ce552702e5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Replace $(UT_APP) in nvme.unittest.mk with $(APP); there is no actual
difference in the way they are used.
Change-Id: I039c2488ee998ee8b1a67af129045b66a97bc29c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The .o files are always kept anyway, so there is no need for an explicit
rule.
Change-Id: Id1687ba89daabfda5802e4328deb127403277928
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
