Multi-Packet Receive Queue is to receive multiple packets on a single large
buffer. The number of consumed strides in CQE is accumulated to keep track
of the current stride index. However, it is safer to directly use stride
index in CQE to avoid out-of-order situation which can possibly be caused
by introducing LRO in the future.
If Rx CQE compression is enabled, HW can be configured to store the stride
index in a mini-CQE but this will need newer version of library/driver.
Therefore, since this change, MPRQ is only supported with the newer
library/driver and Rx hash result is not supported if MPRQ is enabled along
with Rx CQE compression.
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Shahaf Shuler <shahafs@mellanox.com>
If Multi-Packet RQ is enabled but not supported by device or
kernel/library, print out a warning message.
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Stride index is added to mlx5_mini_cqe8 structure and WQE ID is added to
mlx5_cqe structure.
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Shahaf Shuler <shahafs@mellanox.com>
mlx5_rx_poll_len() returns Rx hash result extracted from either mini CQE or
regular CQE. As mini CQE may not have the hash result if configured
otherwise, it shouldn't assume the first DWORD of mini CQE is always hash
result. mlx5_rx_poll_len() is changed to return pointer to the mini CQE if
compressed.
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Shahaf Shuler <shahafs@mellanox.com>
The threshold of buffer replenishment for vectorized Rx burst is a constant
value (64). If the size of Rx queue is comparatively small, device could
run out of buffers. For example, if the size of Rx queue is 128, buffers
are replenished only twice per a wraparound. This can cause jitter in
receiving packets and the jitter can cause unnecessary retransmission for
TCP connections.
Fixes: 6cb559d67b83 ("net/mlx5: add vectorized Rx/Tx burst for x86")
Fixes: 570acdb1da8a ("net/mlx5: add vectorized Rx/Tx burst for ARM")
Cc: stable@dpdk.org
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Shahaf Shuler <shahafs@mellanox.com>
The generic tunnel TSO was depended in the regular one capabilities to
be enabled.
Cc: stable@dpdk.org
Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
rte_errno should be saved only if error has occurred because rte_errno
could have garbage value.
Fixes: a6d83b6a9209 ("net/mlx5: standardize on negative errno values")
Cc: stable@dpdk.org
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Split maintainers logs from user logs.
A lot of debug logs are present providing internal information on how
the PMD works to users. Such logs should not be available for them and
thus should remain available only when the PMD is compiled in debug
mode.
This commits removes some useless debug logs, move the Maintainers ones
under DEBUG and also move dump into debug mode only.
Cc: stable@dpdk.org
Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
The mlx5 driver had two init functions, but this could
cause log initialization to be done after the
other initialization. Also, the name of the function does
not match convention (cut/paste error?).
Fix by initializing log type first at start of the pmd_init.
This also gets rid of having two constructor functions.
Fixes: a170a30d22a8 ("net/mlx5: use dynamic logging")
Cc: stable@dpdk.org
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
This artificial limitation was inherited from the mlx4 code base and has no
purpose other than adding unnecessary noise.
This patch is a port of commit f2318196c71a ("net/mlx4: remove limitation
on number of instances").
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
The callback should be invoked only for memory that has been registered
in a device, hence, no need to track cleanup events if no device is
present.
Bugzilla ID: 56
Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support")
Signed-off-by: David Marchand <david.marchand@6wind.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
Tested-by: Ferruh Yigit <ferruh.yigit@intel.com>
MR cache init takes place on the device configuration.
When the device is re-configured multiple times, for example when
changing the number of queue on the flight, deadlock can happen.
This patch moved MR cache init from device configuration function to
probe function to make sure init only once.
Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support")
Signed-off-by: Xueming Li <xuemingl@mellanox.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
Although uncommon, applications may destroy a device immediately after
probing it without going through dev_configure() first.
This patch addresses a crash which occurs when mlx5_dev_close() calls
mlx5_mr_release() due to an uninitialized entry in the private structure.
Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support")
Signed-off-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
On some distros, the inbox rdma-core tree can contain the Software
Parser enum while the remaining structs still missing.
Fixes: 5f8ba81c4228 ("net/mlx5: support generic tunnel offloading")
Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
This is to fix the offloads introduced by commits
5f8ba81 net/mlx5: support generic tunnel offloading
5355f44 ethdev: introduce generic IP/UDP tunnel checksum and TSO
Fixes: 8589e944d075 ("net/mlx5: fix setting offsets for SW parser")
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Since ConnectX-5, SW parser just complements HW parser. SW parser starts to
engage only if HW parser can't reach a header. For the older devices, HW
parser will not kick in if any of SWP offsets is set. Therefore, all of the
L3 offsets should be set regardless of HW offload. As IPv6 doesn't have
header checksum, the mbuf can't have PKT_TX_[OUTER_]IP_CKSUM if outer or
inner L3 is IPv6.
And if inner packet isn't IP, the inner offsets shouldn't be set.
Fixes: 5f8ba81c4228 ("net/mlx5: support generic tunnel offloading")
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Xueming Li <xuemingl@mellanox.com>
With the commit af4f09f28294 ("net/mlx5: prefix all functions with mlx5"),
mlx5_xstats_get() is not compliant any longer with the api.
It always returns the caller max entries count while it should return how
many entries it wrote/wanted to write.
Fixes: af4f09f28294 ("net/mlx5: prefix all functions with mlx5")
Signed-off-by: David Marchand <david.marchand@6wind.com>
Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
This patch adds a pair of "()" to embrace the argument
input to the function-like macro invocation.
drivers/net/mlx5/mlx5_rxtx_vec.c:37:
drivers/net/mlx5/mlx5_rxtx_vec_neon.h:170:24: error: too many arguments
provided to function-like macro invocation
(uint16x8_t) { 0, 0, cs_flags, rte_cpu_to_be_16(len),
Fixes: 570acdb1da ("net/mlx5: add vectorized Rx/Tx burst for ARM")
Cc: stable@dpdk.org
Signed-off-by: Gavin Hu <gavin.hu@arm.com>
Signed-off-by: Sirshak Das <sirshak.das@arm.com>
Reviewed-by: Phil Yang <phil.yang@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Reviewed-by: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
Using inner RSS by default for GRE leads to memory corruption as the
extra flow items added for the inner RSS are not counted in the flow
attributes buffer size.
Fixing by enforcing the default RSS level to be outer. This much
simplify the flow engine and more robust.
Future optimization for out of the box RSS can be done on subsequent
commits.
Fixes: d4a405186b73 ("net/mlx5: support tunnel RSS level")
Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
IBV_RX_HASH_INNER should be referenced only when having tunnel support
in the Verbs headers.
Fixes: 80f2d0ed7ff9 ("net/mlx5: add hardware flow debug dump")
Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
Drop flow rules are created on the ETH queue even though the parser layer
matches the flow rule layer (L3/L4)
Fixes: 6f2f4948b236 ("net/mlx5: fix flow director rule deletion crash")
Cc: stable@dpdk.org
Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
Acked-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
Fixes: ccdcba53a3f4 ("net/mlx5: use Netlink to add/remove MAC addresses")
Signed-off-by: Andy Green <andy@warmcat.com>
Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
When a multi-segmented packet is inlined, data can be further inlined even
after the first segment. In case of TSO packet, extra inline data after TSO
header should be carried by an inline DSEG which has 4B inline header
recording the length of the inline data. If more than one segment is
inlined, the length doesn't count from the second segment. This will cause
a fault in HW and CQE will have an error, which is ignored by PMD.
Fixes: f895536be4fa ("net/mlx5: enable inlining data from multiple segments")
Cc: stable@dpdk.org
Signed-off-by: Xueming Li <xuemingl@mellanox.com>
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
A new hook function is added and called inside the PMDs at the end
of the device probing:
- in primary process, after allocating, init and config
- in secondary process, after attaching and local init
This new function is almost empty for now.
It will be used later to add some post-initialization processing.
For the PMDs calling the helpers rte_eth_dev_create() or
rte_eth_dev_pci_generic_probe(), the hook rte_eth_dev_probing_finish()
is called from here, and not in the PMD itself.
Note that the helper rte_eth_dev_create() could be used more,
especially for vdevs, avoiding some code duplication in PMDs.
Cc: stable@dpdk.org
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Reviewed-by: Stephen Hemminger <stephen@networkplumber.org>
Multi-Packet Rx Queue (MPRQ a.k.a Striding RQ) can further save PCIe
bandwidth by posting a single large buffer for multiple packets. Instead of
posting a buffer per a packet, one large buffer is posted in order to
receive multiple packets on the buffer. A MPRQ buffer consists of multiple
fixed-size strides and each stride receives one packet.
Rx packet is mem-copied to a user-provided mbuf if the size of Rx packet is
comparatively small, or PMD attaches the Rx packet to the mbuf by external
buffer attachment - rte_pktmbuf_attach_extbuf(). A mempool for external
buffers will be allocated and managed by PMD.
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Shahaf Shuler <shahafs@mellanox.com>
Filling in fields of mbuf becomes a separate inline function so that this
can be reused.
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Shahaf Shuler <shahafs@mellanox.com>
This is the new design of Memory Region (MR) for mlx PMD, in order to:
- Accommodate the new memory hotplug model.
- Support non-contiguous Mempool.
There are multiple layers for MR search.
L0 is to look up the last-hit entry which is pointed by mr_ctrl->mru (Most
Recently Used). If L0 misses, L1 is to look up the address in a fixed-sized
array by linear search. L0/L1 is in an inline function -
mlx5_mr_lookup_cache().
If L1 misses, the bottom-half function is called to look up the address
from the bigger local cache of the queue. This is L2 - mlx5_mr_addr2mr_bh()
and it is not an inline function. Data structure for L2 is the Binary Tree.
If L2 misses, the search falls into the slowest path which takes locks in
order to access global device cache (priv->mr.cache) which is also a B-tree
and caches the original MR list (priv->mr.mr_list) of the device. Unless
the global cache is overflowed, it is all-inclusive of the MR list. This is
L3 - mlx5_mr_lookup_dev(). The size of the L3 cache table is limited and
can't be expanded on the fly due to deadlock. Refer to the comments in the
code for the details - mr_lookup_dev(). If L3 is overflowed, the list will
have to be searched directly bypassing the cache although it is slower.
If L3 misses, a new MR for the address should be created -
mlx5_mr_create(). When it creates a new MR, it tries to register adjacent
memsegs as much as possible which are virtually contiguous around the
address. This must take two locks - memory_hotplug_lock and
priv->mr.rwlock. Due to memory_hotplug_lock, there can't be any
allocation/free of memory inside.
In the free callback of the memory hotplug event, freed space is searched
from the MR list and corresponding bits are cleared from the bitmap of MRs.
This can fragment a MR and the MR will have multiple search entries in the
caches. Once there's a change by the event, the global cache must be
rebuilt and all the per-queue caches will be flushed as well. If memory is
frequently freed in run-time, that may cause jitter on dataplane processing
in the worst case by incurring MR cache flush and rebuild. But, it would be
the least probable scenario.
To guarantee the most optimal performance, it is highly recommended to use
an EAL option - '--socket-mem'. Then, the reserved memory will be pinned
and won't be freed dynamically. And it is also recommended to configure
per-lcore cache of Mempool. Even though there're many MRs for a device or
MRs are highly fragmented, the cache of Mempool will be much helpful to
reduce misses on per-queue caches anyway.
'--legacy-mem' is also supported.
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
This patch removes current support of Memory Region (MR) in order to
accommodate the dynamic memory hotplug patch. This patch can be compiled
but traffic can't flow and HW will raise faults. Subsequent patches will
add new MR support.
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
This patch check if a input requested offloading is valid or not.
Any reuqested offloading must be supported in the device capabilities.
Any offloading is disabled by default if it is not set in the parameter
dev_conf->[rt]xmode.offloads to rte_eth_dev_configure() and
[rt]x_conf->offloads to rte_eth_[rt]x_queue_setup().
If any offloading is enabled in rte_eth_dev_configure() by application,
it is enabled on all queues no matter whether it is per-queue or
per-port type and no matter whether it is set or cleared in
[rt]x_conf->offloads to rte_eth_[rt]x_queue_setup().
If a per-queue offloading hasn't be enabled in rte_eth_dev_configure(),
it can be enabled or disabled for individual queue in
ret_eth_[rt]x_queue_setup().
A new added offloading is the one which hasn't been enabled in
rte_eth_dev_configure() and is reuqested to be enabled in
rte_eth_[rt]x_queue_setup(), it must be per-queue type,
otherwise trigger an error log.
The underlying PMD must be aware that the requested offloadings
to PMD specific queue_setup() function only carries those
new added offloadings of per-queue type.
This patch can make above such checking in a common way in rte_ethdev
layer to avoid same checking in underlying PMD.
This patch assumes that all PMDs in 18.05-rc2 have already
converted to offload API defined in 17.11 . It also assumes
that all PMDs can return correct offloading capabilities
in rte_eth_dev_infos_get().
In the beginning of [rt]x_queue_setup() of underlying PMD,
add offloads = [rt]xconf->offloads |
dev->data->dev_conf.[rt]xmode.offloads; to keep same as offload API
defined in 17.11 to avoid upper application broken due to offload
API change.
PMD can use the info that input [rt]xconf->offloads only carry
the new added per-queue offloads to do some optimization or some
code change on base of this patch.
Signed-off-by: Wei Dai <wei.dai@intel.com>
Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
rte_eth_devices[] is not shared between primary and secondary process, but
a static array to each process. The reverse pointer of device (priv->dev)
is invalid. Instead, priv has the pointer to shared data of the device,
struct rte_eth_dev_data *dev_data;
Two macros are added,
#define PORT_ID(priv) ((priv)->dev_data->port_id)
#define ETH_DEV(priv) (&rte_eth_devices[PORT_ID(priv)])
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
rdma-core doesn't add up max_tso_header size to max_inline_data size. The
library takes bigger value between the two.
Fixes: 43e9d9794cde ("net/mlx5: support upstream rdma-core")
Cc: stable@dpdk.org
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Acked-by: Shahaf Shuler <shahafs@mellanox.com>
If something went wrong in mlx5_pci_prob the allocated eth dev
will cause a memory leak.
This commit release the eth dev that was previously allocated.
Fixes: 771fa900b73a ("mlx5: introduce new driver for Mellanox ConnectX-4 adapters")
Cc: stable@dpdk.org
Signed-off-by: Raslan Darawsheh <rasland@mellanox.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
When attr_ctx is NULL it will attempt to free the list of devices twice.
Avoid double freeing the list by directly going to error handling.
Fixes: 771fa900b73a ("mlx5: introduce new driver for Mellanox ConnectX-4 adapters")
Cc: stable@dpdk.org
Signed-off-by: Raslan Darawsheh <rasland@mellanox.com>
Acked-by: Yongseok Koh <yskoh@mellanox.com>
Item spec and last are wrongly compared to the NIC capability causing a
validation failure when the mask is null.
This validation function should only verify the user is not configuring
unsupported matching fields.
Fixes: 2097d0d1e2cc ("net/mlx5: support basic flow items and actions")
Cc: stable@dpdk.org
Signed-off-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
mlx5 prefixed function returns a negative errno value.
the error handler on mlx5_pci_probe is doing the same.
Fixes: a6d83b6a9209 ("net/mlx5: standardize on negative errno values")
Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
Upon success, mlx5_socket_connect should return the fd descriptor of the
primary process
Fixes: a6d83b6a9209 ("net/mlx5: standardize on negative errno values")
Signed-off-by: Yongseok Koh <yskoh@mellanox.com>
Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
A new ethdev API was exposed by
commit 3be82f5cc5e3 ("ethdev: support PMD-tuned Tx/Rx parameters")
Enabling the PMD to provide default parameters in case no strict request
from application in order to improve the out of the box experience.
While the current API lacks the means for the PMD to provide the best
possible value, providing the best default the PMD can guess.
The values are based on Mellanox performance report and depends on the
underlying NIC capabilities.
Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
According to ethtool_link_setting API recommendation ETHTOOL_GLINKSETTINGS
should be called before ETHTOOL_GSET as the later one deprecated.
Fixes: f47ba80080ab ("net/mlx5: remove kernel version check")
Signed-off-by: Shahaf Shuler <shahafs@mellanox.com>
Acked-by: Nelio Laranjeiro <nelio.laranjeiro@6wind.com>
