In some E-Switch use cases, applications want to receive all traffic
on a single port. Since currently, flow API does not provide a way to
match traffic forwarded to any port representor, this patch adds
support for controlling representor matching on ingress flow rules.
Representor matching is controlled through a new device argument
repr_matching_en.
- If representor matching is enabled (default setting),
then each ingress pattern template has an implicit REPRESENTED_PORT
item added. Flow rules based on this pattern template will match
the vport associated with the port on which the rule is created.
- If representor matching is disabled, then there will be no implicit
item added. As a result ingress flow rules will match traffic
coming to any port, not only the port on which the flow rule is
created.
Representor matching is enabled by default, to provide an expected
default behavior.
This patch enables egress flow rules on representors when E-Switch is
enabled in the following configurations:
- repr_matching_en=1 and dv_xmeta_en=4
- repr_matching_en=1 and dv_xmeta_en=0
- repr_matching_en=0 and dv_xmeta_en=0
When representor matching is enabled, the following logic is
implemented:
1. Creating an egress template table in group 0 for each port. These
tables will hold default flow rules defined as follows:
pattern SQ
actions MODIFY_FIELD (set available bits in REG_C_0 to
vport_meta_tag)
MODIFY_FIELD (copy REG_A to REG_C_1, only when
dv_xmeta_en == 4)
JUMP (group 1)
2. Egress pattern templates created by an application have an implicit
MLX5_RTE_FLOW_ITEM_TYPE_TAG item prepended to the pattern, which
matches available bits of REG_C_0.
3. Egress flow rules created by an application have an implicit
MLX5_RTE_FLOW_ITEM_TYPE_TAG item prepended to the pattern, which
matches vport_meta_tag placed in available bits of REG_C_0.
4. Egress template tables created by an application, which are in
group n, are placed in group n + 1.
5. Items and actions related to META are operating on REG_A when
dv_xmeta_en == 0 or REG_C_1 when dv_xmeta_en == 4.
When representor matching is disabled and extended metadata is disabled,
no changes to the current logic are required.
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
This patch adds support for fdb_def_rule_en device argument to HW
Steering, which controls:
- the creation of the default FDB jump flow rule.
- the ability of the user to create transfer flow rules in the root
table.
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Signed-off-by: Xueming Li <xuemingl@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
The queue based rte_flow_async_action_* functions work the same as
queue based async flow functions. The operations can be pushed
asynchronously, and so is the pull.
This commit adds the async action missing push and pull support.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Add support for AGE action for HW steering.
This patch includes:
1. Add new structures to manage aging.
2. Initialize all of them in configure function.
3. Implement per second aging check using CNT background thread.
4. Enable AGE action in flow create/destroy operations.
5. Implement a queue-based function to report aged flow rules.
Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Add the ability to create an indirect action handle for METER_MARK.
It allows sharing one Meter between several different actions.
Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
This commit adds the support of connection tracking to HW steering as
SW steering did before.
The difference from SW steering implementation is that it takes
advantage of HW steering bulk action allocation support, in HW
steering only one single CT pool is needed.
An indexed pool is introduced to record allocated actions from bulk and
CT action state etc. Once one CT action is allocated from bulk, one
indexed object will also be allocated from the indexed pool, similar to
deallocating. That makes mlx5_aso_ct_action can also be managed by that
indexed pool, no need to be reserved from mlx5_aso_ct_pool. The single
CT pool is also saved to mlx5_aso_ct_action struct directly.
The ASO operation functions are shared with SW steering implementation.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
This patch adapts mlx5 PMD to changes in mlx5dr API regarding the action
templates. It changes the following:
1. Actions template creation:
- Flow actions types are translated to mlx5dr action types in order
to create mlx5dr_action_template object.
- An offset is assigned to each flow action. This offset is used to
predetermine the action's location in the rule_acts array passed
on the rule creation.
2. Template table creation:
- Fixed actions are created and put in the rule_acts cache using
predetermined offsets
- mlx5dr matcher is parametrized by action templates bound to
template table.
- mlx5dr matcher is configured to optimize rule creation based on
passed rule indices.
3. Flow rule creation:
- mlx5dr rule is parametrized by the action template on which these
rule's actions are based.
- Rule index hint is provided to mlx5dr.
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
This commit adds HW steering counter action support.
The pool mechanism is the basic data structure for the HW steering
counter.
The HW steering's counter pool is based on the rte_ring of zero-copy
variation.
There are two global rte_rings:
1. free_list:
Store the counters indexes, which are ready for use.
2. wait_reset_list:
Store the counters indexes, which are just freed from the user and
need to query the hardware counter to get the reset value before
this counter can be reused again.
The counter pool also supports cache per HW steering's queues, which are
also based on the rte_ring of zero-copy variation.
The cache can be configured in size, preload, threshold, and fetch size,
they are all exposed via device args.
The main operations of the counter pool are as follows:
- Get one counter from the pool:
1. The user call _get_* API.
2. If the cache is enabled, dequeue one counter index from the local
cache:
2. A: if the dequeued one from the local cache is still in reset
status (counter's query_gen_when_free is equal to pool's query
gen):
I. Flush all counters in the local cache back to global
wait_reset_list.
II. Fetch _fetch_sz_ counters into the cache from the global
free list.
III. Fetch one counter from the cache.
3. If the cache is empty, fetch _fetch_sz_ counters from the global
free list into the cache and fetch one counter from the cache.
- Free one counter into the pool:
1. The user calls _put_* API.
2. Put the counter into the local cache.
3. If the local cache is full:
A: Write back all counters above _threshold_ into the global
wait_reset_list.
B: Also, write back this counter into the global wait_reset_list.
When the local cache is disabled, _get_/_put_ cache directly from/into
global list.
Signed-off-by: Xiaoyu Min <jackmin@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
This commit adds meter action for HWS steering.
HW steering meter is based on ASO. The number of meters will
be used by flows should be specified in advance in the flow
configure API.
Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
The new mode 4 of devarg "dv_xmeta_en" is added for HWS only. In this
mode, the Rx / Tx metadata with 32b width copy between FDB and NIC is
supported.
The mark is only supported in NIC and there is no copy supported.
Signed-off-by: Bing Zhao <bingz@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
This patch implements creating and caching of port action for use with
HW Steering FDB flows.
Actions are created on flow template API configuration and created
only on the port designated as the master. Attaching and detaching ports
in the same switching domain causes an update to the port actions cache
by, respectively, creating and destroying actions.
A new devarg fdb_def_rule_en is being added and it's used to control
the default dedicated E-Switch rules that are created by the PMD
implicitly or not, and PMD sets this value to 1 by default.
If set to 0, the default E-Switch rule will not be created and the user
can create the specific E-Switch rules on the root table if needed.
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
This patch introduces support for modify_field rte_flow actions in HWS
mode that includes:
- Ingress and egress domains,
- SET and ADD operations,
- usage of arbitrary bit offsets and widths for packet and metadata
fields.
This is implemented in two phases:
1. On flow table creation the hardware commands are generated, based
on rte_flow action templates, and stored alongside action template.
2. On flow rule creation/queueing the hardware commands are updated with
values provided by the user. Any masks over immediate values, provided
in action templates, are applied to these values before enqueueing rules
for creation.
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
As the rte_flow_async API defines the action mask with a field value
not being 0 means the action will be used as shared in all the flows
in the table.
The header reformat action with the action mask field not being 0 will
be created as constant shared action. For encapsulation header reformat
action, there are two kinds of encapsulation data, raw_encap_data
and rte_flow_item encap_data. Both of these two kinds of data can be
identified from the action mask conf as constant or not.
Examples:
1. VXLAN encap (encap_data: rte_flow_item)
action conf (eth/ipv4/udp/vxlan_hdr)
a. action mask conf (eth/ipv4/udp/vxlan_hdr)
- items are constant.
b. action mask conf (NULL)
- items will change.
2. RAW encap (encap_data: raw)
action conf (raw_data)
a. action mask conf (not NULL)
- encap_data constant.
b. action mask conf (NULL)
- encap_data will change.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
In the flow_dv_hashfields_set() function, while item_flags was 0,
the code went directly to the first if and the else case would
never have a chance to be checked. This caused the IPv6 and TCP hash
fields in the else case would never be set.
This commit adds the dedicated HW steering hash field set function
to generate the RSS hash fields.
Fixes: 3a2f674b6a ("net/mlx5: add queue and RSS HW steering action")
Cc: stable@dpdk.org
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
In the function flow_get_drv_type(), attr will be read in non-HWS mode.
In case the user calls the HWS API in SWS mode, the attr should be
placed in HWS functions or it will cause a crash.
Fixes: c40c061a02 ("net/mlx5: add basic flow queue operation")
Cc: stable@dpdk.org
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
The debug layer is used to generate a debug CSV file
containing details of the context, table, matcher, rules
and other useful debug information.
Signed-off-by: Hamdan Igbaria <hamdani@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
Action objects are used for executing different HW actions
over packets. Each action contains the HW resources and parameters
needed for action use over the HW when creating a rule.
Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
HWS rule objects reside under the matcher, each rule holds
the configuration for the packet fields to match and the
set of actions to execute over the packet that has the requested
fields.
Rules can be created asynchronously in parallel over multiple
queues to different matchers with each rule configured to the HW.
Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
HWS matcher resides under the table object, each table can
have multiple chained matches with different attributes.
Each matcher represents a combination of match and action templates,
and can contain multiple configurations based on the templates.
Packets are steered from the table to the matcher and from there to
other objects.
The matcher allows efficient HW packet field matching and action
execution based on the configuration done to it.
Signed-off-by: Alex Vesker <valex@nvidia.com>
Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
HWS table resides under the context object, each context can
have multiple tables with different steering types RX/TX/FDB.
The table is not only a logical object but it is also represented
in the HW, packets can be steered to the table, and from there
to other tables.
Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
Context is the first mlx5dr object created, all sub objects
table, matcher, rule, and action are created using the context.
The context holds the capabilities and the send queues used for
configuring the offloads to the HW.
Signed-off-by: Alex Vesker <valex@nvidia.com>
Definers are HW objects that are used for matching, rte items
are translated to definers, each definer holds the fields and
bit-masks used for HW flow matching. The definer layer is used
for finding the most efficient definer for each set of items.
In addition to definer creation we also calculate the field
copy (fc) array used for efficient items to WQE conversion.
Signed-off-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
HWS configures flows to the HW using a QP, each WQE has
the details of the flow we want to offload. The send layer
allocates the resources needed to send the request to the HW
as well as managing the queues, getting completions and
handling failures.
Signed-off-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
HWS needs to manage different types of device memory in
an efficient and quick way. For this, memory pools are
being used.
Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
This adds the command layer which is used to communicate with
the FW, to query capabilities and allocate FW resources needed
for HWS.
Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
Add missing glue support for the HWS mlx5dr layer.
The new glue functions are needed for mlx5dv to create
matcher and action, which are used as the kernel root
table as well as for capabilities query like device
name and port info.
Signed-off-by: Alex Vesker <valex@nvidia.com>
This stores the available tags that can be used by the
application in a global array that will be used to
transfer the TAG item directly from the ID to the REG_C_x
since these can't be changed after startup.
Signed-off-by: Bing Zhao <bingz@nvidia.com>
Add new fields to flow table capabilities to query the capabilities to
set, add, and copy to REG_C_x.
The set capability is queried and saved for future usage.
Signed-off-by: Bing Zhao <bingz@nvidia.com>
This adds conversion functions between both ethdev port_id and IB
context to internal corresponding tag/mask values.
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
This provides shared item tranlsation code for hardware
steering root table flows as they still work under FW steering mode.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
This split the item matcher and value translation to
make the code reusable for the new steering mode.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
This splits flow item translation code to a dedicated function
to share the item translation code with hardware steering mode.
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Introduce mlx5_get_send_to_kernel_priority() function which returns
value of priority which must be used to jump back to table 0 in order
to send traffic to kernel. This function returns lowest priority.
Add flow_dv_translate_action_send_to_kernel() function which
will allocate rdma-core send_to_kernel action object.
Called from flow_dv_translate().
Fail translation of RTE_FLOW_ACTION_TYPE_SEND_TO_KERNEL action in
HW steering.
Signed-off-by: Michael Savisko <michaelsav@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Add new structure mlx5_send_to_kernel_action which will hold
together allocated action resource and a reference to used table.
A new structure member of this type added to struct mlx5_dev_ctx_shared.
The member will be initialized upon first created send_to_kernel
action and will be reused for all future actions of this type.
Release of these resources will be done when all shared DR
resources are being released in mlx5_os_free_shared_dr().
Change function flow_dv_tbl_resource_release() from
static to external.
Signed-off-by: Michael Savisko <michaelsav@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Add new mlx5 action flag MLX5_FLOW_ACTION_SEND_TO_KERNEL.
Add element MLX5_FLOW_FATE_SEND_TO_KERNEL in enum mlx5_flow_fate_type.
For that purpose field 'fate_action' in structure mlx5_flow_handle must be
expanded from 3 bits to 4 bits.
Signed-off-by: Michael Savisko <michaelsav@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
The thread workspace push/pop should be paired. In the "flow_list_create"
routine, if error happened the workspace pop was missed. This patch shares
the workspace pop for all return paths.
Fixes: 0064bf4318 ("net/mlx5: fix nested flow creation")
Cc: stable@dpdk.org
Signed-off-by: Dong Zhou <dongzhou@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
In order to perform things like LTO more easily in our DPDK applications,
we use DPDK as a meson subproject.
Export include directories list in order to be usable in this context.
Signed-off-by: Markus Theil <markus.theil@tu-ilmenau.de>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Makes apps configurable from meson, like already possible for drivers.
Signed-off-by: Markus Theil <markus.theil@tu-ilmenau.de>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
This function is not defined, remove its declaration.
Fixes: c1f86306a0 ("virtio: add new driver")
Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
This patch promotes the per-queue stats API to stable.
The API has been used by the Vhost PMD since v22.07, and
David Marchand posted a patch to make use of it in next
OVS release[0].
[0]: http://patchwork.ozlabs.org/project/openvswitch/patch/20221007111613.1695524-4-david.marchand@redhat.com/
Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Acked-by: David Marchand <david.marchand@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
When the packet receiving failure and the DMA ring full occur
simultaneously in the asynchronous vhost, the slot_idx needs to be
decreased by 1. For packed virtqueue, the slot index should be
ring_size - 1, if the slot_idx is currently 0, since the ring size is
not necessarily the power of 2.
Fixes: 84d5204310 ("vhost: support async dequeue for split ring")
Fixes: fe8477ebbd ("vhost: support async packed ring dequeue")
Cc: stable@dpdk.org
Signed-off-by: Cheng Jiang <cheng1.jiang@intel.com>
Tested-by: Wei Ling <weix.ling@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
When vhost receive packets from the front-end using packed virtqueue, it
might use multiple descriptors for one packet, so we need calculate and
record the descriptor number for each packet to update available
descriptor counter and used descriptor counter, and rollback when DMA
ring is full.
Fixes: fe8477ebbd ("vhost: support async packed ring dequeue")
Cc: stable@dpdk.org
Signed-off-by: Cheng Jiang <cheng1.jiang@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
Vhost-user library locks all VQ's access lock when processing
vring based messages, such as SET_VRING_KICK and SET_VRING_CALL,
and the data processing thread may already be started, e.g: SPDK
vhost-blk and vhost-scsi will start the data processing thread
when one vring is ready, then deadlock may happen when SPDK is
posting interrupts to VM. Here, we add a new API which allows
caller to try again later for this case.
Bugzilla ID: 1015
Fixes: c573699830 ("vhost: fix missing virtqueue lock protection")
Cc: stable@dpdk.org
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
Add the queue id and/or the port id in some logs, so it is easier to
understand what happens.
Signed-off-by: Olivier Matz <olivier.matz@6wind.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
This patch applies rte_vhost_async_dma_unconfigure() to manually free
DMA vChannels. Before unconfiguration, make sure the specified DMA
vChannel is no longer used by any vhost ports.
Signed-off-by: Xuan Ding <xuan.ding@intel.com>
Tested-by: Wei Ling <weix.ling@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
Add a new API rte_vhost_async_dma_unconfigure() to unconfigure DMA
vChannels in vhost async data path. Lock protection are also added
to protect DMA vChannel configuration and unconfiguration
from concurrent calls.
Signed-off-by: Xuan Ding <xuan.ding@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
