When using Intel C++ compiler(icc) 14.0.1.106 or the older icc 13.x
version, the mbuf initializer variable was not getting configured
correctly, as the mb_def variable was not set correctly. This is due
to an issue with icc (DPD200249565 which already been fixed in
icc 14.0.2 and newer compiler release) where it incorrectly calculates
the field offsets with initializers when zero-sized fields
are used in a structure.
To work around this, the code in ixgbe_rxq_vec_setup does not setup the
fields using an initializer, but instead assigns the values individually
in code.
NOTE: There is no performance impact to this change as the queue
setup functions are not data-plane APIs, but are only used at app
initialization.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: David Marchand <david.marchand@6wind.com>
Acked-by: Thomas Monjalon <thomas.monjalon@6wind.com>
Remove the "static" prefix to the template mbuf variable in
ixgbe_rxq_vec_setup function. This will then allow different
threads to initialize different RX queues at the same time,
without one overwriting the other's data.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Changchun Ouyang <Changchun.ouyang@intel.com>
An error has been introduced by commit 1f22652ca8
("fix perf regression due to moved pool ptr").
Fix the case where RTE_MBUF_REFCNT is disabled.
Signed-off-by: Changchun Ouyang <changchun.ouyang@intel.com>
Acked-by: Thomas Monjalon <thomas.monjalon@6wind.com>
Clang fails with an error about a variable being used uninitialized:
lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c:67:30:
error: variable 'dma_addr0' is uninitialized
when used here [-Werror,-Wuninitialized]
dma_addr0 = _mm_xor_si128(dma_addr0, dma_addr0);
^~~~~~~~~
This error can be fixed by replacing the call to xor which
takes two parameters, by a call to setzero, which does not take any.
Reported-by: Keith Wiles <keith.wiles@windriver.com>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Thomas Monjalon <thomas.monjalon@6wind.com>
Since the introduction of vector PMD, a bug in ixgbe_rxq_rearm could
cause a crash. As long as the memory pool allocated to the RX queue
has mbufs available, there is no problem. After allocation of _all_
mbufs from the memory pool, previously returned mbufs by
rte_eth_rx_burst could be accessed by subsequent calls to the PMD and
could be returned by subsequent calls to rte_eth_rx_burst. From the
perspective of the application, the means that fields within the mbuf
could change and that previously allocated mbufs could appear multiple
times.
After failure of mbuf allocation, the dd bits should indicate that the
packets are not ready. For this, this patch adds code to reset the dd
bits in the first RTE_IXGBE_DESCS_PER_LOOP packets of the next
RTE_IXGBE_RXQ_REARM_THRESH packets only if the next
RTE_IXGBE_RXQ_REARM_THRESH packets that will be accessed contain
previously allocated packets.
Setting the bits is not enough. The bits are checked _after_ setting
the mbuf fields, thus a mechanism is needed to prevent the previously
used mbuf pointers from being accessed during the speculative load of
the mbuf fields. For this reason, not only the dd bits are reset, but
also the mbufs associated to those descriptors are set to point to a
"fake" mbuf.
Signed-off-by: Balazs Nemeth <balazs.nemeth@intel.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
The refcnt field is contained within an anonymous union within the mbuf
data structure, and gcc 4.4 gives an error about an unknown field unless
the initialiser for the field is contained within extra braces.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Thomas Monjalon <thomas.monjalon@6wind.com>
Provide a wrapper routine to enable receive of scattered packets with a
vector driver. This improves the performance of the slow-path RX.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Adjust the fast-path code to fix the regression caused by the pool
pointer moving to the second cache line. This change adjusts the
prefetching and also the way in which the mbufs are freed back to the
mempool.
Note: slow-path e.g. path supporting jumbo frames, is still slower, but
is dealt with by a later commit
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
The vector PMD expects fields to be in a specific order so that it can
do vector operations on multiple fields at a time. Following mbuf
rework, adjust driver to take account of the new layout and re-enable it
in the config.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
The rte_pktmbuf structure was initially included in the rte_mbuf
structure. This was needed when there was 2 types of mbuf (ctrl and
packet). As the control mbuf has been removed, we can merge the
rte_pktmbuf into the rte_mbuf structure.
Advantages of doing this:
- the access to mbuf fields is easier (ex: m->data instead of m->pkt.data)
- make the structure more consistent: for instance, there was no reason
to have the ol_flags field in rte_mbuf
- it will allow a deeper reorganization of the rte_mbuf structure in the
next commits, allowing to gain several bytes in it
Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
[Bruce: updated for latest code and new example apps]
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
It seems that RTE_MBUF_SCATTER_GATHER is not the proper name for the
feature it provides. "Scatter gather" means that data is stored using
several buffers. RTE_MBUF_REFCNT seems to be a better name for that
feature as it provides a reference counter for mbufs.
The macro RTE_MBUF_SCATTER_GATHER is poisoned to ensure this
modification is seen by drivers or applications using it.
Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Since all unspecified fields in an initializer are assumed to be zero we
can simplify the empty mbuf definition in the vector driver to only use
the fields that are non-zero, i.e. just nb_segs = 1. This makes things
shorter and means that the structure doesn't need as many updates for
other fields being renamed or moved.
The variable itself is never modified and only used by a single function
so it can be made const and local to the using function.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Pablo de Lara <pablo.de.lara.guarch@intel.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
When writing to the mbuf array for receiving packets, do not assume
16-byte alignment by using aligned stores. If the pointers are only
8-byte aligned, the program will crash due to incorrect alignment.
Changing "store" to "storeu" fixes this.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Thomas Monjalon <thomas.monjalon@6wind.com>
ixgbe was failing to build in the default configuration because it required
sse4.2 intrinsics, and the default config doesn't support more than sse3.
Modify the pmd so that only sse3 intrinsics are pulled in and used.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: Konstantin Ananyev <konstantin.ananyev@intel.com>
CC: Thomas Monjalon <thomas.monjalon@6wind.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
RTE_IXGBE_RX_OLFLAGS_ENABLE gives a hint whick keeping packet type
in RX ol_flags or not.
By default it is set to update ol_flags in RX mbuf header.
If unset it, will gain addtional performance, but will lose packet
type information.
Signed-off-by: Cunming Liang <cunming.liang@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Yong Liu <yong.liu@intel.com>
Tested-by: Zhaochen Zhan <zhaochen.zhan@intel.com>
Vectorized Rx cannot be enabled if RTE_LIBRTE_IEEE1588=y.
So variable dev is not used in this case.
Signed-off-by: Cunming Liang <cunming.liang@intel.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Yong Liu <yong.liu@intel.com>
Tested-by: Zhaochen Zhan <zhaochen.zhan@intel.com>
New file containing optimized receive and transmit functions which
use 128bit vector instructions to improve performance. When conditions
permit, these functions will be enabled at runtime by the device
initialization routines already in the PMD.
The compilation of the vectorized RX and TX code paths is controlled by
a new setting in the build time configuration for the IXGBE driver. Also
added is a setting which allows an optional further performance increase
by disabling the use of the olflags field on packet RX.
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Tested-by: XiaonanX Zhang <xiaonanx.zhang@intel.com>
Tested-by: Waterman Cao <waterman.cao@intel.com>
[Thomas: code-style adjustments]
Acked-by: Thomas Monjalon <thomas.monjalon@6wind.com>
