This allows comparing the Linux kernel driver's performance to the SPDK
user-mode NVMe driver.
Change-Id: I71c70163a4133c2f237c8c57b3c698ec261455f5
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename the nvme_free_request macro to nvme_dealloc_request to match
nvme_alloc_request and add a wrapper function to nvme.c so that the
macro contents are only expanded once.
The DPDK nvme_impl.h uses rte_mempool_put(), which generates a large
amount of code inline. Moving this macro expansion to a wrapper
function avoids inlining it in the multiple places nvme_free_request()
gets called, most of which are error handling cases that are not in the
hot I/O path.
Change-Id: I64ea9c39ba47e26672eee8d5058f1489e07eee5b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The I/O splitting tests don't pass the I/Os through the normal
completion path, so we need to free the children ourselves.
Fixes all Valgrind warnings for nvme_ns_cmd_ut.
Change-Id: Iaf7d9f7f4cab71428a0123ee30d0f6042001e423
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows the same objects to be linked into static and shared
libraries and allows the creation of position-independent executables
with the static libraries.
Change-Id: I119949c3644c02a83e414227615dcc2d8f896286
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Some of these are covered under -Wall, but add them all just to be clear.
Change-Id: I63104be4bf58becd19a4c30a4d275d2c24c6761d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Modern toolchains already enable noexecstack by default, but set it
explicitly just to be sure.
Change-Id: I2866220db5a5ed0bb6e78a4e83763839ea3b93b0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This forces the ELF loader to resolve all relocations immediately at
load time rather than allowing lazy binding.
Therefore the loader can make the GOT (global offset table) read-only
once the relocations are complete during the loading process.
Change-Id: I89a878d1060ce79d58c5be7700d8b122a33a8ceb
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is generated by the CUnit library when built with
--enable-memtrace.
In particular, the CUnit package on FreeBSD is configured this way.
Change-Id: I616b7d91ed560be2706089287510a772a5c47e11
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Like sprintf() with automatic buffer allocation.
This should help to avoid fixed-size buffers in
non-performance-sensitive code that formats strings.
Change-Id: I35209ae84014ed5daf41baa5b03af8a5f6b02b8e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_ut is testing multi-threaded operations, and Valgrind's thread
behavior is different than running the program natively. Under
Valgrind, the unit test essentially hangs waiting for the global
variable to be updated.
Change-Id: Id3665002c16ac3e695c50325375305a76f72cee4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
nvme_alloc_request() does not zero out the request (this is
intentional, since the real implementation uses a mempool where requests
get reused). Add nvme_allocate_request() wrapper that initializes the
request correctly. This fixes all uses of uninitialized memory caught
by Valgrind in nvme_qpair_ut. This test was also failing in practice on
FreeBSD due to non-zero buffers returned from malloc().
Change-Id: I2d6ea29289bd4887aaa9120fba6bce10088e6917
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The default version of nvme_impl.h was cleaned up to release the pthread
mutex attr on failure cases, but the unit test version did not receive
this update.
Change-Id: I899b7dc809393dc8e6fd24ed98e1d0a61ecf1c95
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Check the LBA and block count fields of the I/Os generated by the
splitting function to ensure they were split correctly.
Change-Id: I84abb1ac462fb7423d51a1be384fd1be68dfecae
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Instead of repeating the check for conditions that must be fulfilled to
continue the tests, just use CU_ASSERT_FATAL, which will abort the test
and return.
Change-Id: If617b286a587d9efb1ce57b90061634ed5bc7ae8
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This is the only place nvme_request_add_child() is used, so move it
nearby and make it static to allow the compiler to inline it.
Change-Id: If4a7e17fde0b0272e1d4432c1dcedbec27c25371
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The current implementation of nvme_assert in the unit test nvme_impl.h
just prints the message and continues.
We should not be triggering assert conditions, even in the unit test
code, so make nvme_assert actually call assert(). This lets us catch
mistakes in the unit tests more easily.
Also fix the two unit tests that currently trigger an assert:
- The I/O splitting test in nvme_ns_cmd_ut was passing an invalid
combination of NULL payload with non-zero lba_count.
- The ctrlr_cmd test was passing an invalid number of entries to
nvme_ctrlr_cmd_get_error_page(). This case should probably not be an
assert but rather an error code. However, the function does not
return a status code currently, so it is not trivial to make that
change. For now, just drop the asserting test case and the code added
to the test to work around it.
While we're here, fix the macros in the unit test nvme_impl.h so they
are usable in single-line conditionals without braces - that is the
whole point of the do { ... } while (0) pattern, so there should be no
trailing semicolon.
Change-Id: Iad503c5c5d19a426d48c80d9a7d6da12ff2c982a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Cover more functions and conditions for nvme_ctrlr_cmd.
Remove unnecessary lines in nvme_ctrlr_ut.
Update nvme_qpair_ut.
Remove unnecessary header file.
Change-Id: I8c5a75573b26210ca57711b366acd55ab96614c0
Signed-off-by: Liang Yan <liangx.yan@intel.com>
Move dependency includes into a new spdk.deps.mk file,
then include it at the end of Makefiles that build
source files.
Also add a test to autobuild.sh to confirm that
binaries are regenerated if we make after touching a
header file.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If6a1905706a840f92cbdf3ace7fbdb27fe2de213
Exercise both the DEBUG=y path (during autobuild) and DEBUG=n (during
autopackage).
Change-Id: Ie2f0fea42ac675cd18f49d0b3274aa2c3413b88a
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Set -DNDEBUG to make assert() a no-op. Since nvme_assert() is just an
alias for assert(), this controls compilation of nvme_assert as well.
Change-Id: Ie9d0c7fea007eff030179c1b54865adc3dcaf6b6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Pull the almost-identical request splitting code for driver-assisted
striping and maximum I/O size into its own function,
_nvme_ns_cmd_split_request().
Change-Id: I3c15ac2073f8f5aec721c427199c8fb1a5d6a1fc
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Initialize the full nvme_namespace structure in prepare_for_test() so
that e.g. ns->id is not used uninitialized.
Also check for request allocation failure - if the request is NULL, we
can't run the rest of the tests that dereference request without
crashing.
Change-Id: I3010ca3e81f153a4d0201498a14a963c2b9e960d
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This helps enable FreeBSD, where pciaccess pci_device_has_kernel_driver()
is not functional. The function will return 0 if there is no driver
attached, or the Linux uio or FreeBSD nic_uio driver is attached. It will
return 1 otherwise.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I0921e61c9040b1e0411b5dc40b36fc7f2721c8c5
This will rmmod nvme on Linux, and use the DPDK nic_uio driver
to unbind nvme devices from the nvme driver on FreeBSD.
"nic_uio" is actually a misnomer - it is not NIC driver specific,
and can safely be used to bind NVMe devices to an effectively
null driver.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8b964a070586166d762d79696385b82eeb2e88bd
Then only include this new file on Linux, to help enable building
SPDK on FreeBSD.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Id818a2d8662c95e603d8b580f1f4b92889cc626e
The changes are minor:
- remove unneeded error.h
- replace PATH_MAX with a suitable local #define
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5aecf8b53e0ac7582f394c71b4668888a6c6292f
The Linux pagemap-based implementation obviously does not
work on FreeBSD. DPDK has data structures describing the huge
pages it has allocated, so use that instead when we need to
populate new 2MB mappings in our tables.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I924e104f42891aaa2f931159aabba2779f239e91
This will add different flags, src files, etc. for FreeBSD.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I4f2d664cf42d01745318f241ce9558dccf148b7d
GCC generates a series of 64-bit MOV instructions for the memcpy() into
the submission queue. We can do better with 128-bit SSE2 instructions.
DPDK already has a memcpy implementation that is optimized for small
inline copies, so use it instead of memcpy.
Change-Id: I5f09259b4d5cb089ace4a8ea6d2078c03fee84f3
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
No change in behavior, just a simplification.
We already have a check for retry, so pull the cb_fn check out and put
it under the !retry branch.
This makes it clearer that requests that are going to be retried will
not get their callbacks called.
Change-Id: I70c7067e550c7fca78b0441b5474833f73863315
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
