Makes the code slightly more readable.
Change-Id: Iebf8fb07bceacf433d4bdad0a30419a3faab7eee
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/439370
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
We use those values in various places in SPDK,
so let's define them in a single place now.
Change-Id: Iad9a5745d69166a6e6032370d4e5a0e604914e45
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/439369
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
This follows the same trend as the mem_map APIs.
Currently, most of the spdk_vtophys() callers manually
detect physically noncontiguous buffers to split them
into multiple physically contiguous chunks. This patch
is a first step towards encapsulating most of that logic
in a single place - in spdk_vtophys() itself.
This patch doesn't change any functionality on its own,
it only extends the API.
Change-Id: I16faa9dea270c370f2a814cd399f59055b5ccc3d
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/438449
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: wuzhouhui <wuzhouhui@kingsoft.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
As a part of cleanup they're replaced by a device-agnostic
attach API, which is easier for us to manage.
Change-Id: Ia92bd8f4525712bd27ade16ead67435c5e0fbe7a
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/436479
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
The static method ioat_channel_destruct() never fails. This cleanup patch
changes its return type to be void instead of int.
Change-Id: I0db464b6a810eadc3f58550114c4532ca33d03b5
Signed-off-by: Rami Rosen <ramirose@gmail.com>
Reviewed-on: https://review.gerrithub.io/425936
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
The method spdk_ioat_detach() never fails. This patch changes its
return type to be void instead of int.
Change-Id: I6f5bda9ab330c02ae0535bed8b78d416c7b1e264
Signed-off-by: Rami Rosen <ramirose@gmail.com>
Reviewed-on: https://review.gerrithub.io/425252
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
I/OAT device will refuse to resume from the Halted
state until we clear both CHANERR and CHANERR_INT
registers.
We only cleared CHANERR so far. So after the I/OAT
device encountered an error, SPDK would not be able
to initialize it ever again unless it's rebound to
the ioatdma driver.
Unlike CHANERR, CHANERR_INT is only accessible through
PCI config space.
CHANERR_INT is only available on I/OAT devices with
version < 3.3.
Change-Id: Ib369ca76f58c4868fe61ff9532b2e7947e5091d3
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/422026
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
The IOAT "ring" is actually just a circular linked list of descriptors;
the descriptors do not need to be in a single physically contiguous
region. This can be accomodated by calling spdk_vtophys() on each
descriptor rather than assuming they are all in a single contiguous
region.
Also store the physical address of each descriptor in its associated
software descriptor context to avoid the need to call spdk_vtophys()
during runtime.
Change-Id: Ic8636bbc61deb496a0c6d0ea56b75d298f5f426c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/417782
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Change-Id: I9e7ff07de8b06afb45b1059d455c8fe854035640
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/417783
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
This code was created when spdk_pci_device wasn't
available yet. Now we can use spdk_pci_device* instead
of void* for extra code clarity.
Change-Id: I81d440720b22a484ae3d2739e0510a021bebbafe
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/416995
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Pawel Wodkowski <pawelx.wodkowski@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
This was obviously omitted in the original implementation.
Change-Id: Ifba049cdb8d1c1f24e30adb542946a2fa83c7464
Signed-off-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/416655
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Automatically detect more whitespace errors.
All existing cases are fixed; only whitespace change (verify with
diff -w) except for one comment style fixup in include/spdk/nvme.h.
Change-Id: If750e54b9c8e3421ea6feda5f20184a31431631e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/402360
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Require braces around all conditional statements, e.g.:
if (cond)
statement();
becomes:
if (cond) {
statement();
}
This is the style used through most of the SPDK code, but several
exceptions crept in over time. Add the astyle option to make sure we
are consistent.
Change-Id: I5a71980147fe8dfb471ff42e8bc06db2124a1a7f
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/390914
Reviewed-by: <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Dariusz Stojaczyk <dariuszx.stojaczyk@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Disambiguate the log components from the trace functionality
(include/spdk/trace.h).
The internal spdk_trace_flag structure and related functions will be
renamed in a later commit - this is just a find and replace on
SPDK_TRACE_* and SPDK_LOG_REGISTER_TRACE_FLAG().
Change-Id: I617bd5a9fbe35ffb44ae6020b292658c094a0ad6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/376421
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
This code was still using an old paradigm of returning the
number of bytes associated with a successful submission.
Just return 0 on success instead - if caller needs the
number of bytes for some reason they have the information
to get it.
While here, return an appropriate negated errno where possible -
we especially want ENOMEM returned when an ioat channel runs out
of descriptors.
Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I5858ccd6cff916b6c80fda7d2c9fce96fb39ef89
Reviewed-on: https://review.gerrithub.io/378858
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
- rename spdk_malloc_socket to spdk_dma_malloc_socket
- rename spdk_malloc to spdk_dma_malloc
- rename spdk_zmalloc to spdk_dma_zmalloc
- rename spdk_realloc to spdk_dma_realloc
- rename spdk_free to spdk_dma_free
Change-Id: I52a11b7a4243281f9c56f503e826fd7c4a1fd883
Signed-off-by: John Meneghini <johnm@netapp.com>
Reviewed-on: https://review.gerrithub.io/362604
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
These were repeated a few different places, so pull them into a common
header file.
Change-Id: Id807fa2cfec0de2e0363aeb081510fb801781985
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Change the PCI enumeration API to individual functions per device type
so that only the drivers that are actually in use get linked into the
final executable. All of the common code is still shared internally in
the env_dpdk library.
Change-Id: I2ba83afe59202a510f999a0674e23e60b6581221
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The SPDK_TRACELOG macro depends on a CONFIG setting (DEBUG), so it
should not be part of the public API.
Create a new include/spdk_internal directory for headers that should
only be used within SPDK, not exported for public use.
Change-Id: I39b90ce57da3270e735ba32210c4b3a3468c460b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This allows users to swap their PCI library from
libpciaccess/dpdk to another mechanism using the standard
method for swapping out the env library.
Change-Id: Ib2248f8b43754a540de2ec01897e571f0302b667
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
This patch also drops support for automatically unbinding
devices from the kernel - run scripts/setup.sh first.
Our generic pci interface is now hidden behind include/spdk/pci.h
and implemented in lib/util/pci.c. We no longer wrap the calls
in nvme_impl.h or ioat_impl.h. The implementation now only uses
DPDK and the libpciaccess dependency has been removed. If using
a version of DPDK earlier than 16.07, enumerating devices
by class code isn't available and only Intel SSDs will be
discovered. DPDK 16.07 adds enumeration by class code and all
NVMe devices will be correctly discovered.
Change-Id: I0e8bac36b5ca57df604a2b310c47342c67dc9f3c
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
assert is part of the C standard library and is available
on any platform we'd consider porting to. Don't put a
wrapper around it.
Change-Id: I0acfdd6a8a269d6c37df38fb7ddf4f1227630223
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
pthreads are widely supported and are available on any
platform we currently foresee porting to. Use that API
instead of attempting to abstract it away to simplify
the code.
Change-Id: I28123d427ea8da07c6329b0233f0702f2d85c2a0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Add a parameter to each I/OAT library function that requires a channel
instead of implicitly using the thread-local channel registration model.
I/OAT channels are already reported by the spdk_ioat_probe() attach
callback, so no infrastructure for channel allocation is necessary.
Change-Id: I8731126fcaea9fe2bafc41a3f75c969a100ef8f0
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rename all functions with a spdk_ prefix, and provide enough of an API
to avoid apps needing to #include <pciaccess.h>.
The opaque type used in the public API for a PCI device is now
struct spdk_pci_device *.
Change-Id: I1e7a09bbc5328c624bec8cf5c8a69ab0ea8e8254
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Similar to the NVMe API change, this allows better abstraction of the
PCI subsystem.
Change-Id: I2b84d9c3c498a08d4451b4ff27d0865f0456c210
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
For those Crystal Beach DMA channels which support block fill capability,
we add a fill API here that can zero out pages or fill them with a
fixed pattern.
Change-Id: I8a57337702b951c703d494004b111f6d206279fb
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Also add a space between Copyright and (c).
The copyright year can be determined using git metadata.
Also remove the duplicated "All rights reserved." - every instance of
this line already has a corresponding "All rights reserved" immediately
below it, except for examples/ioat/kperf/kmod/dma_perf.c, where I have
added it manually.
Performed using this command:
git ls-files | xargs sed -i -e 's/Copyright(c) \(.*\) Intel Corporation. All rights reserved./Copyright (c) Intel Corporation./'
Change-Id: I3779f404966800709024eb1eb66a50068af2716c
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This ensures that any uninitialized fields are 0/NULL so if
ioat_channel_start() fails, ioat_channel_destruct() will not try to free
bogus pointers.
Change-Id: I99278c9fa280cbcdf3f7448e77db3ac98b59cdd6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The ioat library currently only supports DMA copy operations, but the
hardware can do other types of transfers. Add a union of the hardware
descriptor structures to enable support for the other operations in the
future.
Also add a generic hardware descriptor type to allow access to the parts
of the descriptor that are common between all types.
Change-Id: I3b54421ce771f58b78910e790b53026f311f918e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
This lets us signal an error if the channel is halted in
ioat_process_channel_events().
Change-Id: Iffaf4fd1e27d1254f9d95a37d732ae4a5f3a0465
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Rather than individually allocating each ring entry, use two large
allocations, one for the hardware descriptors and one for the software
descriptor contexts.
This allows the use of simple array indexing on the rings and also
allows the removal of most of the software descriptor structure,
since the necessary information can be retrieved based on the ring
index now.
Change-Id: I73ef24450f69ca0fc35e350286282c6b1c77a207
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
NVMe doesn't require the specific 64-bit MMIO ordering on 32-bit
platforms performed in spdk_mmio_read_8(), but it doesn't hurt.
We have to pick one of the two possible orderings, so pick the one
required by I/OAT.
Change-Id: I2b909d64d0c077b797d0f64a11d78d1ecc55eec7
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
The ioat driver supports DMA engine copy offload hardware available on
Intel Xeon platforms.
Change-Id: Ida0b17b25816576948ddb1b0443587e0f09574d4
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
