Commit Graph

242 Commits

Author SHA1 Message Date
Jiewei Ke
a9bdb1ee73 lib/nvmf: mask pending AENs
Per section 5.2 of the NVMe 1.3 spec:

"When the controller posts a completion queue entry for an outstanding
Asynchronous Event Request command and thus reports an asynchronous
event, subsequent events of that event type are automatically masked by
the controller until the host clears that event. An event is cleared by
reading the log page associated with that event using the Get Log Page
command (see section 5.14)."

Signed-off-by: Jiewei Ke <jiewei@smartx.com>
Change-Id: I0773ce1b704c2124db354fd19e0a8a19da48da54
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5462
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2020-12-18 09:48:07 +00:00
Jiewei Ke
8cd9ef2825 lib/nvmf: Support to queue pending async events
Add support to queue pending async events to avoid event lost if
multiple async events happen when there is no outstanding AER requests.
Pending async events will be consumes in FIFO order once any AER
requests are received.

Signed-off-by: Jiewei Ke <jiewei@smartx.com>
Change-Id: I545f0baa4ec2996e9e02ec12c176d639e3c0d55a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5117
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-12-18 09:48:07 +00:00
Jim Harris
3e96796391 nvmf: assert that subsystem cb_fn/cb_arg don't get set twice
nvmf_subsystem_state_change() protects against going
starting the pause/change/resume chain again if we
haven't completed the previous chain.  But add
some asserts to make sure the cb_fn and cb_args
never get set twice.  This also requires setting
them back to NULL after they've been used.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I56c6c89bfb108223f731d5ac28100abbb746ab2f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5531
Community-CI: Broadcom CI
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-12-14 14:32:29 +00:00
Shuhei Matsumoto
5c9a8a3c41 lib/nvmf: Use qpair_get_listen_trid() to find a subsystem listener
There is a public API spdk_nvmf_qpair_get_listen_trid() and it can
get trid of the specified qpair safely for any transport including
pluggable transports.

The API was overlooked when implementing the multipath feature for
NVMe-oF target. trid pointer was added to struct spdk_nvmf_qpair and
was used to find a subsystem listener.

However, pluggable transports got seg. fault because trid of the
qpair was not set.

To avoid such segmentation fault for any transport, change
nvmf_ctrlr_create() to use spdk_nvmf_qpair_get_listen_trid().

The struct spdk_nvmf_qpair is located in the public header file,
and so leave the added trid for now. It will be deprecated eventually.

Reported-by: Jacek Kalwas <jacek.kalwas@intel.com>
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I5e0bd24bd58b6ffdf1352332a179a82682f1589f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5323
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
2020-12-01 10:21:28 +00:00
Ben Walker
b10e305010 nvmf: In spdk_nvmf_request_exec, reverse order of subsystem and qpair
checks

Check if the subsystem is paused before checking whether the qpair is
active. The order of these checks doesn't really matter, but in the next
patch it will be more convenient to check the subsystem first.

Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: Ibc95e2578e9a95296f5fbc6023af3b542e954781
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5015
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-12-01 10:20:43 +00:00
Ben Walker
c642e8e2ee nvmf: Inline _nvmf_request_exec
This function is only called from one spot, so it's simpler to combine
it with that function.

Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: I0a35e11edaf6c4446de530a3b0f538aa4a3238f7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/5014
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2020-12-01 10:20:43 +00:00
Madhu Adav MJ
fce9428772 nvmf: Async event support for discovery log change
Added asynchronous event notices for discovery log change
as per nvme fabrics spec 1.1. This allows a host with persistent
connection to discovery controller to automatically connect to any
new subsystem available to the host automatically.

According to nvme fabrics spec 1.1, if the connect command specifies a
non-zero keep alive timer value and the discovery controller does not
support asynchronous events then we need to return Connect Invalid.
Since SPDK does not implement this check instead added support
for asynchronous events in discovery controller.

Change-Id: I4cade5f7d24826ce97a2fa2b4ca688a1d728c1db
Signed-off-by: Madhu Adav MJ <madhu.adav@flipkart.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4870
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Anil Veerabhadrappa <anil.veerabhadrappa@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Broadcom CI
2020-11-11 00:53:57 +00:00
Michael Haeuptle
4409007906 nvmf: Fixes double triggering of association timer
Fixes issue #1635.

Under rare circumstances, the CC.en and CC.shn are both set
which then results in setting the association timer twice.
This scenario was observed during hot plug testing when the
initiator tries to reset the subsystem that contains the
removed device.
The end result is that when the ctrlr is destructed, then
one of the timers can still fire and access freed memory.

Change-Id: Ie5880ab325a28f19361f73712bdeb5b58894ee68
Signed-off-by: Michael Haeuptle <michael.haeuptle@hpe.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4935
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-10-29 08:11:12 +00:00
yidong0635
fd661859ed nvmf/ctrlr: Fix scanbuild warning.
Issue:
ctrlr.c:1851:13: warning: Although the value stored
to 'copy_len' is used in the enclosing expression,
the value is never actually read from 'copy_len'
        next_pos = copy_len = 0;
                   ^          ~

Signed-off-by: yidong0635 <dongx.yi@intel.com>
Change-Id: Ie6d402cd4ad206ae5713deb15e2d03929bac94f4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4823
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Community-CI: Mellanox Build Bot
2020-10-27 08:42:32 +00:00
Ben Walker
2ec88c4f8c nvmf: Don't fail a property set command with invalid bits
If the initiator sends a property set command to a valid register, but
the value contains bits that are reserved by the NVMe specification,
don't fail the whole command. Just log an error message.

Previously, any valid bits set would take effect but then the command
would also fail. That confuses the initiator and thinks it must retry
the full property set.

Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: I566bc68c4469b9f41c69902e276f825c86683075
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4814
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: John Kariuki <John.K.Kariuki@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-10-23 13:47:35 +00:00
Tomasz Zawadzki
4e8e97c886 log: remove internal log.h header
There is nothing left here, so remove it.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Ib947d42bc577dbebb4650b1be885e05a80f8f8cf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4541
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Anil Veerabhadrappa <anil.veerabhadrappa@broadcom.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Community-CI: Broadcom CI
2020-10-15 08:23:39 +00:00
Tomasz Zawadzki
2172c432cf log: simplify SPDK_LOG_REGISTER_COMPONENT
This patch removes the string from register component.
Removed are all instances in libs or hardcoded in apps.

Starting with this patch literal passed to register,
serves as name for the flag.

All instances of SPDK_LOG_* were replaced with just *
in lowercase.
No actual name change for flags occur in this patch.

Affected are SPDK_LOG_REGISTER_COMPONENT() and
SPDK_*LOG() macros.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I002b232fde57ecf9c6777726b181fc0341f1bb17
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4495
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Mellanox Build Bot
Reviewed-by: Anil Veerabhadrappa <anil.veerabhadrappa@broadcom.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom CI
2020-10-14 08:00:35 +00:00
Ben Walker
5e4e4bc485 nvmf: Group bool members of spdk_nvmf_subsystem into a bit field
This saves 2 bytes and allows it to pack nicely with the
changing state bool (which must remember separate for atomic
operations).

Change-Id: Ibb92ae3c74306e60385ae23d0aaf877f33a69095
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4553
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-10-09 07:18:49 +00:00
Changpeng Liu
778e4c0020 nvmf: make nvmf_ctrlr_process_fabrics_cmd() as a static function
Unlike ADMIN and IO commands, the FABRIC command is only processed
in the ctrlr.c file.

Change-Id: Ic4e01c7f81c98631a2c7cb603343b301f8ba63e1
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4307
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-09-22 11:40:18 +00:00
Shuhei Matsumoto
5cf0c3707f lib/nvmf: Use macro constant for ANA descriptor to avoid variable length array
In C language, we cannot use constant at compile time. Hence the
local array _ana_desc[] is not a fixed size array but a variable
length array.

We can avoid using variable length array by changing const variable
to macro constant.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I7333a8078d3102c4bd5088f56f6530846854c85f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4093
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Monica Kenguva <monica.kenguva@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-09-16 07:55:28 +00:00
Shuhei Matsumoto
523f2a85d3 lib/nvmf: Set up ANA Change Notice
If the ANA reporting feature is enabled for the subsystem,
- set ANA Change Notice of Asynchronous Event Configuration to 1
- set ANA Change Notice of Optional Asynchronus Event Supported to 1
- set ANA Non-Optimized state and ANA Inaccessible state of ANA
  Capability to 1.

ANA Change state is not used and ANA Persistent Loss state is not
supported for now.

The next patch will actually support ANA Change Notice using an new
RPC.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I4db2e33dd2879cdf995adcab41ef53728b27a201
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4087
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-09-16 07:55:28 +00:00
Jacek Kalwas
ae207216fb nvmf: remove request exec backdoor
It was introduced for the purpose of executing fabric cmds when
subsystem and qpairs are not active. It was rather workaround than
solution for transport type like vfio-user. spdk_nvmf_request_exec
is a preferred way of passing request obj into nvmf layer.

Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I4f989de27bfd494c744017599909c2e200f0f233
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4180
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2020-09-14 09:18:49 +00:00
Shuhei Matsumoto
98145aa6b1 lib/nvmf: Replace raw number 0xffffffff by SPDK_NVME_GLOBAL_NS_TAG
For nsid, use SPDK_NVME_GLOBAL_NS_TAG rather than raw number
0xffffffff wherever possible.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I23e989786263172e13bab40c011cf58beb06fabf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4055
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-09-07 09:29:37 +00:00
Shuhei Matsumoto
68f1681771 lib/nvmf: Control I/O and some admin commands according to ANA state
For I/O commands, block them if ANA state is inaccessible, persistent
loss, or change.

For Identify command, clear capacity field (nuse) to 0 if ANA state
is inaccessible or persistent loss.

For Get Features command, block features, error recovery, write
atomicity normal, reservation notification mask, and reservation
persistence if ANA state is inaccessible, persistent loss, or change.

For Get Log Page command, error information page does not return
any data yet, and hence there is no change.

For Set Features command, if ANA state is inaccessible or change,
block the command if NSID is 0xFFFFFFFF or if feature is error recovery,
write atomicity normal, reservation notification mask, or reservation
persistence, or if ANA state is persistent loss, block the command.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I15dd593227e451aa2247c53da42b6acad1757907
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4043
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-09-04 08:14:18 +00:00
Shuhei Matsumoto
4688f0819f lib/nvmf: Add ANA state to struct spdk_nvmf_subsystem_listener
Add ANA state to struct spdk_nvmf_subsystem_listener and initialize
it to optimized.

Then ctrlr->listener->ana_state is referred when creating ANA log page.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I978424e51d3f23ca72dee30192bc2693abfe203d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4012
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-09-04 08:14:18 +00:00
Shuhei Matsumoto
5b27db6268 lib/nvmf: Check if I/O qpair is on the same listener as admin qpair
We will have ANA state per listener and per subsystem. On the other hand,
NVMe specification defines ANA state per controller.

However, it is possible that I/O qpair and admin qpair are different
listeners on a single controller.

Let's check if I/O qpair is on the same listener as admin qpair if
ANA reporting is enabled.

The case that I/O qpair is on a different listener from admin qpair
is not usual and so the purpose of this check is just to guard SPDK
from any unexpected behavior.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Idb8d255de7f998e45a59a120c2ed5803258873f4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4026
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-09-04 08:14:18 +00:00
Shuhei Matsumoto
7bcff376ab lib/nvmf: Add subsystem_listener to struct spdk_nvmf_ctrlr
Find the subsystem listener whose trid matches req->port->trid when
creating a controller, and store it in the controller.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Iea343b8d8ae827b554df2245b67aed113469c592
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4010
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-09-04 08:14:18 +00:00
Shuhei Matsumoto
58da62239e lib/nvmf: Add nvmf_ctrlr_async_event_ana_change_notice() to notice ANA change
The new function () will be used in the following patches.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I788cfb38d75c3f1f64e1754912b776a80f0f1be8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4007
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-09-04 08:14:18 +00:00
Shuhei Matsumoto
6206e46883 lib/nvme: Set ANA transition time value
ANA transition time shall be non-zero if controller supports ANA
reporting. Linux NVMe host sets this value to 10, and we don't
have any reason to change from that.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I61396695dacf47fad40e3cea3311e555729d9e3e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3909
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2020-08-27 08:36:51 +00:00
Changpeng Liu
01b6bd8a92 nvmf: fix the associate timeout value
SPDK poller uses microsecond as the input parameter, so we need to
change the correct value when opts.association_timeout is expressed
by millisecond.

Change-Id: Ia674f0115ea176b998e4c0c70b8ce75b28984701
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3861
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-08-21 08:23:45 +00:00
Shuhei Matsumoto
6f2265734d lib/nvmf: Optionalize ANA reporting feature
After supporting ANA reporting by default, Linux kernel 5.3 reported
error when parsing NVMe ANA log. The newer kernel fixed the issue
but we should optionalize ANA reporting feature to avoid error for
Linux kernel 5.3 or before.

Add a bool variable ana_reporting to struct spdk_nvmf_subsystem
and disable ANA reporting and initialization of related variables
if it is false. We can expose MNAN (Maximum Number of Allowed
Namespaces) even if ANA reporting is disabled. But MNAN is not
required if ANA reporting is disabled. So do not set MNAN if it is
false too.

Add a public API spdk_nvmf_subsystem_set_ana_reporting() to set
ana_reporting by the nvmf_create_subssytem RPC.

The next patch will add ana_reporting to nvmf_create_subsystem RPC.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Icc77773b4c9513daba2f1a9fdaf951d80574f379
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3850
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Monica Kenguva <monica.kenguva@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-08-21 08:22:40 +00:00
Michael Haeuptle
43ad7febb2 lib/nvmf: Fixes stuck subsystem RPC
A subsystem RPC is not transitioned to a paused state when there
are ios outstanding (tracked by subsystem poll group).

In general AERs, are not tracked as outstanding IOs. However,
there are 3 paths in nvmf_ctrlr_async_event_request which do not
adjust the outstanding io count.
If we get into any of these 3 paths, the subsystem pause can hang
forever.

The issue was reproduced with hot plug stress testing under load.
We can get into the second path (SPDK_NVME_ASYNC_EVENT_TYPE_NOTICE)
under these circumstances:
- An AER completion is sent to the initiator due to a namespace change
(e.g. hot remove/add)
- In this case, type is set to SPDK_NVME_ASYNC_EVENT_TYPE_NOTICE
- The initiator sends a new AER admin command, hitting the second path
where we return without adjusting the outstanding ios.

Fixes: 1552
Change-Id: I45f781966cc1e9a601b2305c7985a21154d802e8
Signed-off-by: Michael Haeuptle <michael.haeuptle@hpe.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3854
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: JinYu <jin.yu@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-08-20 09:31:17 +00:00
Seth Howell
0162da7f76 lib/nvmf: add an in_destruct flag to the ctrlr struct
Helps us avoid adding a new I/O qpair while the ctrlr
is being destroyed.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I3bf9318b075125b9d432b885fa9f6f2f44d422d7
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3686
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-08-17 08:28:49 +00:00
Jacek Kalwas
47ce1fe307 nvmf: association timer triggered on reset
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I85e886e8912009ec5761b5cd0e5b5cef87b25d6e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3463
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2020-08-13 07:53:08 +00:00
Jacek Kalwas
c322453ccc nvmf: disable keep alive timer during shutdown
From the time a shutdown is initiated the controller shall disable
Keep Alive timer.

Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: Id499dabce1913b9da2f0b3fd961fdfc8b621afa9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3462
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2020-08-13 07:53:08 +00:00
Jacek Kalwas
71cd42e139 nvmf: association timer triggered on shutdown
After CC.EN transitions to ‘0’ (due to shutdown or reset), the
association between the host and controller shall be preserved for at
least 2 minutes. After this time, the association may be removed if
the controller has not been re-enabled.

Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Change-Id: I4734600067fd4b7306b46f1325fdd5031e81c079
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2984
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2020-08-13 07:53:08 +00:00
Monica Kenguva
332e846fe6 lib/nvmf: implement ANA log page
SPDK NVMe-oF controller creates a ANA group for each namespace,
ANA group ID matches namespace ID, and default ANA state of ANA group
is optimized, and the MNAN field is set equal to the NN field.

If a ANA log page contains multiple ANA group descriptors, it has
one or more descriptors will not be 8 bytes aligned. Hence we create
one descriptor and copy it to the ANA log page at a time.

Change count will be supported later.

Signed-off-by: Monica Kenguva <monica.kenguva@intel.com>
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I56ba6aa78983480caa3dfbf22aefc9aeabfd5405
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2920
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-08-12 07:50:44 +00:00
Seth Howell
d156d2f771 lib/nvmf: don't free ctrlr->qpair_mask early.
There are 2 messages passed between when
_nvmf_ctrlr_free_from_qpair is executed and when
nvmf_ctrlr_destruct is executed. That leaves time
when the controller->qpair_mask is not a valid
pointer, but it is still in the subsystem
controllers list.

The purpose of this patch is to close that hole.

It is part of a larger series aimed at cleaning up
the controller destruct path.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I0c0199c8392ee278f36df56f599beb10e7a46948
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3685
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-08-11 08:34:15 +00:00
Jacek Kalwas
beb468e9c7 nvmf: fix cc.en and csts.rdy setting during shutdown
CC.EN, CSTS.RDY should not be modified during shutdown.
It doesn't make much sense (against nvme spec) and nvmf spec 1.1
doesn't mentioned it (4.6) either.

Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I7014b10b0217db61c3d380d5c0843808e54577cd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3477
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Seth Howell <seth.howell@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-08-04 16:56:22 +00:00
Jacek Kalwas
aedcec8a0c nvmf: disconnecting io qpairs during shutdown handling
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: Iaee05abead7c685989d91f965133ee6b9e221f89
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3476
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Seth Howell <seth.howell@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-08-04 16:56:22 +00:00
Jacek Kalwas
92f29271d5 nvmf: add aqa property rsvd bits validation
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I824544f8b8faa040a5e0a6802c8ff611d3cb3b86
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3249
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
2020-07-31 08:23:30 +00:00
Jacek Kalwas
e5bf2a1966 nvmf: fix aqa property validation
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I4de16c80fda5a39cfcdb15badb0d37acd1a5a21c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3102
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-07-31 08:23:30 +00:00
Michael Haeuptle
62649a7d7b lib/nvmf: Support custom admin cmd handler for abort
The new abort functionality doesn't take custom admin cmd
handlers into account.
This commit allows setting a custom admin cmd handler
for abort that provides the ability to influence the
bdev lookup to which the abort is sent to.

Signed-off-by: Michael Haeuptle <michael.haeuptle@hpe.com>
Change-Id: I3a66c6f863f5ee4d89cb2194dffdc6855945fa8a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3485
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-07-24 07:25:47 +00:00
Shuhei Matsumoto
81437ff6ce lib/nvmf: Remove req_to_abort parameter from nvmf_ctrlr_abort_request()
struct spdk_nvmf_request holds req_to_abort and so passing req_to_abort
separately is not really necessary now. The internal API
nvmf_ctrlr_abort_request() was added at the stage of prototyping.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I9ef2467d6f92422f044650c62a0777b95c0fc1ab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3488
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
2020-07-24 00:22:24 +00:00
Jacek Kalwas
81717da169 nvmf: use nvme print completion
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: If814edb3c99a7ed9f1800a634f293d3645884457
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3241
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-07-17 07:20:21 +00:00
Jacek Kalwas
afefd815ca nvmf: use nvme print command
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I22747fb342d2e50f40a3bae4f930957f755a97fd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3202
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-07-17 07:20:21 +00:00
Shuhei Matsumoto
604b4503c4 lib/nvmf: Add nvmf_transport_qpair_abort_request()
State machine is different among NVMe-oF transports and is
encapsulated to the transport neutral NVMe-oF controller and
NVMe-oF qpair.

To implement abort operation for each NVMe-oF transport,
add a function pointer qpair_abort_request to struct spdk_nvmf_transport_ops
and a stub nvmf_transport_qpair_abort_request() to encapsulate
which transport is used.

The following patches will implement qpair_abort_request for each
transport. Each qpair_abort_request() is responsible to call
spdk_nvmf_request_complete() for the abort request.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I2beac959ed428c5108cf33691226b7fae5cd24d6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3007
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-07-14 07:54:42 +00:00
Shuhei Matsumoto
a8beb79ea3 lib/nvmf: Add nvmf_ctrlr_abort_request() to call nvmf_bdev_ctrlr_abort_cmd() from transport
If nvmf_ctrlr_abort_request() returns SPDK_NVMF_REQUEST_EXEC_STATUS_COMPLETE,
the caller is responsible to call spdk_nvmf_request_complete().

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I6ed5f8611bcacd20385dd925b9b2e29ed2200269
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3006
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
2020-07-14 07:54:42 +00:00
Shuhei Matsumoto
f045d924fc lib/nvmf: Factor out abort operation on the specific qpair into a function
Factor out abort operation on the specific qpair into a helper
function nvmf_qpair_abort_request().

After this refactoring, nvmf_ctrlr_abort_done() calls
_nvmf_request_complete() only if the passed status is zero.
If the passed status is not zero, nvmf_qpair_abort() is responsible
for calling _nvmf_request_complete() instead.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I4828c0e21cc7650210675661d6e1c0fd54c7a2cb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2991
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
2020-07-14 07:54:42 +00:00
Shuhei Matsumoto
9975d4a1d2 lib/nvmf: Always return success and only toggle bit 0 of CDW0 for abort command
Description is not clear but according to the NVMe specification,
always set the completion status to success and differentiate only
the bit 0 of CDW0 between success and failure for abort command.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I0195e72fe1d7fcc2592f47e9dcf92ac56912282c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1965
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-07-14 07:54:42 +00:00
Jacek Kalwas
ed4b7d12ec nvmf: suppress scan-build warning
Logic error     Dereference of null pointer     ctrlr.c
nvmf_ctrlr_async_event_request  1522

Dereference of null pointer is not possible if sgroup obtained using
ctrlr obj. Adding corresponding asserts suppresses the warning.

Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I78b32fadd5449ee9b533f65193c70e55cf9a8f1c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3251
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-07-10 07:30:59 +00:00
Jacek Kalwas
3dff6a3b48 nvmf: add util to get id from ctrlr
Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: If9246be80d9d43863c1f9cc6ee33c1126dc67fd8
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2983
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
2020-07-08 07:55:03 +00:00
Maciej Szwed
eb05cbd677 pollers: Fix pollers to return correct busy status
Poller should return status > 0 when it did some work
(CPU was used for some time) marking its call as busy
CPU time.

Active pollers should return BUSY status only if they
did any meangful work besides checking some conditions
(e.g. processing requests, do some complicated operations).

Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: Id4636a0997489b129cecfe785592cc97b50992ba
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2164
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
2020-07-07 07:29:31 +00:00
Shuhei Matsumoto
9d4751a550 lib/nvme: Fix Abort Command Limit to process as 0's based value
SPDK NVMe driver had processed ACL as 1's based value by mistake,
and SPDK NVMe-oF target sets ACL to 0. Hence If NVMe driver connects
to SPDK NVMe-oF transport, spdk_nvme_ctrlr_cmd_abort() always queued
abort request.

Fix this bug to process ACL as 0's based value in
spdk_nvme_ctrlr_cmd_abort(). Besides, initialize ACL explicitly to
0 in spdk_nvmf_ctrlr_identify_ctrlr() for clarification.

Signed-off-by: Michael Haeuptle <michael.haeuptle@hpe.com>
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: Id4f3a469776cdab88bcc6f41e7893885a7b78d8c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2513
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Michael Haeuptle <michaelhaeuptle@gmail.com>
2020-06-24 08:19:43 +00:00
Shuhei Matsumoto
315d47da4b lib/nvmf: Clean up abort AER request operation
Include completion operation into nvmf_qpair_abort() and rename
it by nvmf_qpair_abort_aer() for clarification. Update unit test
accordingly.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I763cc7d24b979e27e8775f4e69730466a2351bdf
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2712
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
2020-06-10 13:54:40 +00:00