Commit Graph

2500 Commits

Author SHA1 Message Date
Suanming Mou
d9bad05024 net/mlx5: fix flow table and queue routine on Windows
The macro HAVE_MLX5_HWS_SUPPORT was introduced for HWS only. And
HWS was not supported on Windows. So macro HAVE_MLX5_HWS_SUPPORT
should be only around the code which HWS uses, but avoid including
the code block shared by Linux and Windows.

Fixes: 22681deead ("net/mlx5/hws: enable hardware steering")

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-11-10 18:15:48 +01:00
Shun Hao
139f396bc0 net/mlx5: fix match mask for meter flow
There's an issue introduced by the change of splitting item matcher
and value translation, that the matcher mask value for color is not
set correctly in meter policy flow creation.

This patch fixes this by providing the correct color mask.

Fixes: cd4ab74206 ("net/mlx5: split flow item matcher and value translation")
Cc: stable@dpdk.org

Signed-off-by: Shun Hao <shunh@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
2022-11-10 18:15:48 +01:00
Dariusz Sosnowski
c16e10f180 net/mlx5: forbid direction attribute in transfer flow rules
Since [1] flow API forbids usage of direction attributes in transfer
flow rules. This patch adapts mlx5 PMD to this requirement.

From this patch, flow rule validation in mxl5 PMD will reject transfer
flow rules with any of the direction attributes set
(i.e. 'ingress' or 'egress').
As a result flow rule can only have one of 'ingress', 'egress' or
'transfer' attributes set.

This patch also changes the following:

- Control flow rules used in FDB are 'transfer' only.
- Checks which assumed that 'transfer' can be used
  with 'ingress' and 'egress' are reduced to just checking
  for direction attributes, since all attributes are exclusive.
- Flow rules for updating flow_tag are created for both ingress
  and transfer flow rules which have MARK action.
- Moves mlx5_flow_validate_attributes() function from generic flow
  implementation to legacy Verbs flow engine implementation,
  since it was used only there. Function is renamed accordingly.
  Also removes checking if E-Switch uses DV in that
  function, since if legacy Verbs flow engine is used,
  then that is always not the case.

[1] commit bd2a4d4b2e ("ethdev: forbid direction attribute in
                         transfer flow rules")

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
2022-11-10 18:15:47 +01:00
Jiawei Wang
3e13a6ae95 net/mlx5: fix mirror flow validation with ASO action
While the ASO action(AGE, CT) with the sample action in the one
E-switch mirror flow, due to hardware limitation, the ASO action
after the sample action was not supported.

This patch adds the checking for this validation and reject the flows
with aso action after sample.

Fixes: f935ed4b64 ("net/mlx5: support flow hit action for aging")
Cc: stable@dpdk.org

Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-11-10 18:15:47 +01:00
Jiawei Wang
e9de8f33ca net/mlx5: fix source port checking in sample flow rule
The metadata register C value was lost in FDB egress while doing the
flow sampler on ConnectX-5. The FDB direction checking was decided by
the source port in the flow creation. If there's additional port item
was added in the flow match, then the actual source port was changed.

This patch adds the checking for the port id item:
RTE_FLOW_ITEM_TYPE_PORT_ID, RTE_FLOW_ITEM_TYPE_REPRESENTED_PORT,
and RTE_FLOW_ITEM_TYPE_PORT_REPRESENTOR,
then updates FDB egress checking and the source vport metadata
from the port item, also updates the PUSH VLAN, POP VLAN and
flow sampler action validation.

Fixes: 04c0d3f20f ("net/mlx5: fix port matching in sample flow rule")
Fixes: 255b8f86eb ("net/mlx5: fix E-Switch egress mirror flow validation")
Cc: stable@dpdk.org

Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-11-10 18:15:46 +01:00
Shun Hao
e551740696 net/mlx5: fix action flag data type
MLX5_FLOW_ACTION flags are used as uint64_t now, but some old flags
are not defined as 64 bits. So if they are type casted to uint64 after
bitwise operations, the high 32-bit data might be incorrect.

E.g. Currently MLX5_FLOW_ACTION_DROP is defined as 0x1u, when it is used
like:
	(action_flags & ~MLX5_FLOW_ACTION_DROP)
action_flags is uint64_t so (~MLX5_FLOW_ACTION_DROP) will be casted to
uint64_t as well, but its high 32 bits will be all 0s. This will make the
result not as expected.

This patch fixes this by making all action flags definition as 64-bit
data type.

Fixes: 4b7bf3ffb4 ("net/mlx5: support yellow in meter policy validation")
Cc: stable@dpdk.org

Signed-off-by: Shun Hao <shunh@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
2022-11-10 18:15:46 +01:00
Michael Baum
d37435dc3f net/mlx5: assert for enough space in counter rings
There is a by-design assumption in the code that the global counter
rings can contain all the port counters.
So, enqueuing to these global rings should always succeed.

Add assertions to help for debugging this assumption.

In addition, change mlx5_hws_cnt_pool_put() function to return void due
to those assumptions.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Xiaoyu Min <jackmin@nvidia.com>
2022-11-10 18:15:45 +01:00
Michael Baum
77ca194b4e net/mlx5: add assertions in counter get/put for HWS
Add assertions to help debug in case of counter double alloc/free.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Xiaoyu Min <jackmin@nvidia.com>
2022-11-10 18:15:45 +01:00
Michael Baum
2fd25a6d13 net/mlx5: fix counter elements copies for HWS
The __hws_cnt_r2rcpy() function copies elements from one zero-copy ring
to another zero-copy ring in place.
This routine needs to consider the situation that the address was given
by source and destination could be both wrapped.

It uses 4 different "n" local variables to manage it:
 - n:  Number of elements to copy in total.
 - n1: Number of elements to copy from ptr1, it is the minimal value
       from source/dest n1 field.
 - n2: Number of elements to copy from src->ptr1 to dst->ptr2 or from
       src->ptr2 to dst->ptr1, this variable is 0 when both source and
       dest n1 field are equal.
 - n3: Number of elements to copy from src->ptr2 to dst->ptr2.

The function copies the first n1 elements. If n2 isn't zero it copies
more elements and check whether n3 is zero.
This logic is wrong since n3 may be bigger than zero even when n2 is
zero. This scenario is commonly happening in counters when the internal
mlx5 service thread copies elements from the reset ring into the reuse
ring.

This patch changes the function to copy n3 regardless of n2 value.

Fixes: 4d368e1da3 ("net/mlx5: support flow counter action for HWS")

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Xiaoyu Min <jackmin@nvidia.com>
2022-11-10 18:15:45 +01:00
Michael Baum
5b21f92556 net/mlx5: fix counter access for HWS
The HWS counter has 2 different identifiers:
1. Type "cnt_id_t" which represents the counter inside caches and in
   the flow structure. This index cannot be zero and is mostly called
   "cnt_id".
 2. Internal index, the index in counters array with type "uint32_t".
    mostly it is called "iidx".
The second ID is calculated from the first using "mlx5_hws_cnt_iidx()"
function.

When a direct counter is allocated, if the queue cache is not empty, the
counter represented by cnt_id is popped from the cache. This counter may
be invalid according to the query_gen field. Thus, the "iidx" is parsed
from cnt_id and if it is valid, it is used to update the fields of the
counter structure.
When this counter is invalid, all the cache is flashed and new counters
are fetched into the cache. After fetching, another counter represented
by cnt_id is taken from the cache.
Unfortunately, for updating fields like "in_used" or "age_idx", the
function wrongly may use the old "iidx" coming from an invalid cnt_id.

Update the "iidx" in case of an invalid counter popped from the cache.

Fixes: 4d368e1da3 ("net/mlx5: support flow counter action for HWS")

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Xiaoyu Min <jackmin@nvidia.com>
2022-11-10 18:15:44 +01:00
Michael Baum
a94e89e47b net/mlx5: fix race condition in counter pool resizing
Counter management structure has array of counter pools. This array is
invalid in management structure initialization and grows on demand.

The resizing include:
1. Allocate memory for the new size.
2. Copy the existing data to the new memory.
3. Move the pointer to the new memory.
4. Free the old memory.

The third step can be performed before for this function, and compiler
may do that, but another thread might read the pointer before coping and
read invalid data or even crash.

This patch allocates memory for this array once in management structure
initialization and limit the counters number by 16M.

Fixes: 3aa279157f ("net/mlx5: synchronize flow counter pool creation")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
2022-11-10 18:15:44 +01:00
Suanming Mou
d114dbee28 net/mlx5: enable queue flow aging action
As the queue-based aging API has been integrated[1], the flow aging
action support in HWS steering code can be enabled now.

[1]: https://patchwork.dpdk.org/project/dpdk/cover/
20221026214943.3686635-1-michaelba@nvidia.com/

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
2022-11-10 18:15:43 +01:00
Alex Vesker
0bed4ef76f net/mlx5/hws: remove deprecated rte_atomic
The use of rte_atomic functions is deprecated and is not
required in HWS code. HWS refcounts are used only during
control and always under lock.

Fixes: f8c8a6d844 ("net/mlx5/hws: add action object")

Signed-off-by: Alex Vesker <valex@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
2022-11-10 18:15:43 +01:00
Thomas Monjalon
e9cc7c7abc common/mlx5: use build configuration dictionary
A recent commit added an explicit dependency check on common/mlx5.
For consistency, query dpdk_conf instead of the list of common drivers.
The lists *_drivers should be used only for printing.

Fixes: 3df380f617 ("common/mlx5: fix disabling build")

Suggested-by: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
2022-11-14 11:28:49 +01:00
Bing Zhao
ccc6ea5d9c net/mlx5: fix build with recent compilers
With some higher GCC/CLANG version, it is not recommended to use a
structure with a tailing flexible array inside another structure.
Accessing this array may be considered as a risk to corrupt the
following field even if it is by intention.

The error below was observed:

  drivers/net/mlx5/linux/mlx5_ethdev_os.c: In function 'mlx5_get_flag_dropless_rq':
  drivers/net/mlx5/linux/mlx5_ethdev_os.c:1679:42: error:
  invalid use of structure with flexible array member [-Werror=pedantic]
  1679 | struct ethtool_sset_info hdr;
       | ^~~

Changing it to memory dynamic allocation method will help to get
rid of this complain.

Fixes: e848218741 ("net/mlx5: check delay drop settings in kernel driver")
Cc: stable@dpdk.org

Signed-off-by: Bing Zhao <bingz@nvidia.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
2022-10-31 20:02:05 +01:00
Thomas Monjalon
487599f121 common/mlx5: move build config initialization and check
The variable mlx5_config may be used by other mlx5 drivers
and should be always initialized.
By moving its initialization (with configuration file generation),
it is made consistent for Linux and Windows builds.

And the check of mlx5_config in net/mlx5 is moved at the top of
net/mlx5/hws/meson.build so HWS requirements are in the right context.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Tested-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: David Marchand <david.marchand@redhat.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Alex Vesker <valex@nvidia.com>
2022-10-30 15:55:46 +01:00
Thomas Monjalon
3df380f617 common/mlx5: fix disabling build
If the dependency common/mlx5 is explicitly disabled,
but net/mlx5 is not explicitly disabled,
Meson will read the full recipe of net/mlx5
and will fail when accessing a variable from common/mlx5:
drivers/net/mlx5/meson.build:76:4: ERROR: Unknown variable "mlx5_config".

The solution is to stop parsing net/mlx5 if common/mlx5 is disabled.
The deps array must be defined before stopping, in order to automatically
disable the build of net/mlx5 and print the reason.

The same protection is applied to other mlx5 drivers,
so it will allow using the variable mlx5_config in future.

Fixes: 22681deead ("net/mlx5/hws: enable hardware steering")

Reported-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Tested-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Reviewed-by: David Marchand <david.marchand@redhat.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Alex Vesker <valex@nvidia.com>
2022-10-30 15:55:10 +01:00
Tal Shnaiderman
5976328d91 net/mlx5: fix thread termination check on Windows
The mlx5_is_thread_alive function always returns false
(terminated) regardless to the actual thread state.

Fixed to return the correct thread state.

Bugzilla ID: 1089
Fixes: 5d55a494f4 ("net/mlx5: split multi-thread flow handling per OS")
Cc: stable@dpdk.org

Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-30 08:11:21 +01:00
Michael Baum
c2e3b84ec8 net/mlx5: fix null check in devargs parsing
The "mlx5_os_parse_eth_devargs()" function parses the ETH devargs into a
specific structure called "eth_da".
It gets structure called "devargs" as a member of EAL device containing
the relevant information.

When "devargs" structure is invalid, the function avoids parsing it.
However, when it valid but its field "args" is invalid, the function
tries to parse it and dereference to NULL pointer.

This patch adds check to avoid this NULL dereferencing.

Fixes: 919488fbfa ("net/mlx5: support Sub-Function")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
2022-10-26 13:33:45 +02:00
Jiawei Wang
bfa87e21bd net/mlx5: fix tunnel header with IPIP offload
For the flows with multiple tunnel layers and containing
tunnel decap and modify actions, for example:

... / vxlan / eth / ipv4 proto is 4 / end
actions raw_decap / modify_field / ...
(note: proto 4 means we have the IP-over-IP tunnel in VXLAN payload)

We have added the multiple tunnel layers validation rejecting
the flows like above mentioned one.

The hardware supports the above match combination till the inner
IP-over-IP header (not including the last one), both for IP-over-IPv4
and IP-over-IPv6, so we should not blindly reject. Also, for the modify
actions following the decap we should set the layer attributes correctly.

This patch reverts the below code changes to support the match, and
adjusts the layers update in case of decap with outer tunnel header.

Fixes: fa06906a48 ("net/mlx5: fix IPIP multi-tunnel validation")
Cc: stable@dpdk.org

Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:45 +02:00
Sean Zhang
707d5e7d79 net/mlx5: support flow matching on representor ID
Add support for port_representor item, it will match on traffic
originated from representor port specified in the pattern. This item
is supported in FDB steering domain only (in the flow with transfer
attribute).

For example, below flow will redirect the destination of traffic from
ethdev 1 to ethdev 2.

testpmd> ... pattern eth / port_representor port_id is 1 / end actions
represented_port ethdev_port_id 2 / ...

To handle abovementioned item, Tx queue matching is added in the driver,
and the flow will be expanded to number of the Tx queues. If the spec of
port_representor is NULL, the flow will not be expanded and match on
traffic from any representor port.

Signed-off-by: Sean Zhang <xiazhang@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:44 +02:00
Gregory Etelson
31b29e0c7f net/mlx5: fix RSS expansion buffer size
Increase expansion buffer size to accumulate more RSS types.

Fixes: 3f02c7ff68 ("net/mlx5: fix RSS expansion for inner tunnel VLAN")
Cc: stable@dpdk.org

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
2022-10-26 13:33:44 +02:00
Dariusz Sosnowski
9fa7c1cddb net/mlx5: create control flow rules with HWS
This patch adds the creation of control flow rules required to receive
default traffic (based on port configuration) with HWS.

Control flow rules are created on port start and destroyed on port stop.
Handling of destroying these rules was already implemented before that
patch.

Control flow rules are created if and only if flow isolation mode is
disabled and the creation process goes as follows:

- Port configuration is collected into a set of flags. Each flag
  corresponds to a certain Ethernet pattern type, defined by
  mlx5_flow_ctrl_rx_eth_pattern_type enumeration. There is a separate
  flag for VLAN filtering.

- For each possible Ethernet pattern type and:
  - For each possible RSS action configuration:
    - If configuration flags do not match this combination, it is
      omitted.
    - A template table is created using this combination of pattern
      and actions template (templates are fetched from hw_ctrl_rx
      struct stored in the port's private data).
    - Flow rules are created in this table.

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:43 +02:00
Dariusz Sosnowski
483181f7b6 net/mlx5: support device control of representor matching
In some E-Switch use cases, applications want to receive all traffic
on a single port. Since currently, flow API does not provide a way to
match traffic forwarded to any port representor, this patch adds
support for controlling representor matching on ingress flow rules.

Representor matching is controlled through a new device argument
repr_matching_en.

- If representor matching is enabled (default setting),
  then each ingress pattern template has an implicit REPRESENTED_PORT
  item added. Flow rules based on this pattern template will match
  the vport associated with the port on which the rule is created.
- If representor matching is disabled, then there will be no implicit
  item added. As a result ingress flow rules will match traffic
  coming to any port, not only the port on which the flow rule is
  created.

Representor matching is enabled by default, to provide an expected
default behavior.

This patch enables egress flow rules on representors when E-Switch is
enabled in the following configurations:

- repr_matching_en=1 and dv_xmeta_en=4
- repr_matching_en=1 and dv_xmeta_en=0
- repr_matching_en=0 and dv_xmeta_en=0

When representor matching is enabled, the following logic is
implemented:

1. Creating an egress template table in group 0 for each port. These
   tables will hold default flow rules defined as follows:

      pattern SQ
      actions MODIFY_FIELD (set available bits in REG_C_0 to
                            vport_meta_tag)
              MODIFY_FIELD (copy REG_A to REG_C_1, only when
                            dv_xmeta_en == 4)
              JUMP (group 1)

2. Egress pattern templates created by an application have an implicit
   MLX5_RTE_FLOW_ITEM_TYPE_TAG item prepended to the pattern, which
   matches available bits of REG_C_0.

3. Egress flow rules created by an application have an implicit
   MLX5_RTE_FLOW_ITEM_TYPE_TAG item prepended to the pattern, which
   matches vport_meta_tag placed in available bits of REG_C_0.

4. Egress template tables created by an application, which are in
   group n, are placed in group n + 1.

5. Items and actions related to META are operating on REG_A when
   dv_xmeta_en == 0 or REG_C_1 when dv_xmeta_en == 4.

When representor matching is disabled and extended metadata is disabled,
no changes to the current logic are required.

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:43 +02:00
Dariusz Sosnowski
26e1eaf2da net/mlx5: support device control for E-Switch default rule
This patch adds support for fdb_def_rule_en device argument to HW
Steering, which controls:

- the creation of the default FDB jump flow rule.
- the ability of the user to create transfer flow rules in the root
table.

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Signed-off-by: Xueming Li <xuemingl@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:43 +02:00
Gregory Etelson
a3778a4784 net/mlx5: support flow integrity in HWS group 0
- Reformat flow integrity item translation for HWS code.
- Support flow integrity bits in HWS group 0.
- Update integrity item translation to match positive semantics only.

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:42 +02:00
Suanming Mou
478ba4bbe6 net/mlx5: support async flow action push and pull
The queue based rte_flow_async_action_* functions work the same as
queue based async flow functions. The operations can be pushed
asynchronously, and so is the pull.

This commit adds the async action missing push and pull support.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:42 +02:00
Michael Baum
04a4de756e net/mlx5: support flow age action with HWS
Add support for AGE action for HW steering.
This patch includes:

 1. Add new structures to manage aging.
 2. Initialize all of them in configure function.
 3. Implement per second aging check using CNT background thread.
 4. Enable AGE action in flow create/destroy operations.
 5. Implement a queue-based function to report aged flow rules.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:41 +02:00
Alexander Kozyrev
48fbb0e93d net/mlx5: support flow meter mark indirect action with HWS
Add the ability to create an indirect action handle for METER_MARK.
It allows sharing one Meter between several different actions.

Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:41 +02:00
Gregory Etelson
773ca0e91b net/mlx5: support VLAN push/pop/modify with HWS
Add PMD implementation for HW steering VLAN push, pop, and modify flow
actions.

HWS VLAN push flow action is triggered by a sequence of mandatory
OF_PUSH_VLAN, OF_SET_VLAN_VID, and optional OF_SET_VLAN_PCP
flow action commands.

The commands must be arranged in the exact order:
OF_PUSH_VLAN / OF_SET_VLAN_VID [ / OF_SET_VLAN_PCP ].

In masked HWS VLAN push flow action template *ALL* the above flow
actions must be masked.

In non-masked HWS VLAN push flow action template *ALL* the above flow
actions must not be masked.

Example:

flow actions_template <port id> create \
actions_template_id <action id> \
template \
  of_push_vlan / \
  of_set_vlan_vid \
  [ / of_set_vlan_pcp  ] / end \
mask \
  of_push_vlan ethertype 0 / \
  of_set_vlan_vid vlan_vid 0 \
  [ / of_set_vlan_pcp vlan_pcp 0 ] / end\

flow actions_template <port id> create \
actions_template_id <action id> \
template \
  of_push_vlan ethertype <E>/ \
  of_set_vlan_vid vlan_vid <VID>\
  [ / of_set_vlan_pcp  <PCP>] / end \
mask \
  of_push_vlan ethertype <type != 0> / \
  of_set_vlan_vid vlan_vid <vid_mask != 0>\
  [ / of_set_vlan_pcp vlan_pcp <pcp_mask != 0> ] / end\

HWS VLAN pop flow action is triggered by OF_POP_VLAN
flow action command.
HWS VLAN pop action template is always non-masked.

Example:

flow actions_template <port id> create \
actions_template_id <action id> \
template of_pop_vlan / end mask of_pop_vlan / end

HWS VLAN VID modify flow action is triggered by a standalone
OF_SET_VLAN_VID flow action command.
HWS VLAN VID modify action template can be ether masked or non-masked.

Example:

flow actions_template <port id> create \
actions_template_id <action id> \
template of_set_vlan_vid / end mask of_set_vlan_vid vlan_vid 0 / end

flow actions_template <port id> create \
actions_template_id <action id> \
template of_set_vlan_vid vlan_vid 0x101 / end \
mask of_set_vlan_vid vlan_vid 0xffff / end

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:41 +02:00
Suanming Mou
463170a7c9 net/mlx5: support connection tracking with HWS
This commit adds the support of connection tracking to HW steering as
SW steering did before.

The difference from SW steering implementation is that it takes
advantage of HW steering bulk action allocation support, in HW
steering only one single CT pool is needed.

An indexed pool is introduced to record allocated actions from bulk and
CT action state etc. Once one CT action is allocated from bulk, one
indexed object will also be allocated from the indexed pool, similar to
deallocating. That makes mlx5_aso_ct_action can also be managed by that
indexed pool, no need to be reserved from mlx5_aso_ct_pool. The single
CT pool is also saved to mlx5_aso_ct_action struct directly.

The ASO operation functions are shared with SW steering implementation.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:40 +02:00
Dariusz Sosnowski
f1fecffa88 net/mlx5: support Direct Rules action template API
This patch adapts mlx5 PMD to changes in mlx5dr API regarding the action
templates. It changes the following:

1. Actions template creation:

    - Flow actions types are translated to mlx5dr action types in order
      to create mlx5dr_action_template object.
    - An offset is assigned to each flow action. This offset is used to
      predetermine the action's location in the rule_acts array passed
      on the rule creation.

2. Template table creation:

    - Fixed actions are created and put in the rule_acts cache using
      predetermined offsets
    - mlx5dr matcher is parametrized by action templates bound to
      template table.
    - mlx5dr matcher is configured to optimize rule creation based on
      passed rule indices.

3. Flow rule creation:

    - mlx5dr rule is parametrized by the action template on which these
      rule's actions are based.
    - Rule index hint is provided to mlx5dr.

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:40 +02:00
Xiaoyu Min
4d368e1da3 net/mlx5: support flow counter action for HWS
This commit adds HW steering counter action support.
The pool mechanism is the basic data structure for the HW steering
counter.

The HW steering's counter pool is based on the rte_ring of zero-copy
variation.

There are two global rte_rings:
1. free_list:
     Store the counters indexes, which are ready for use.
2. wait_reset_list:
     Store the counters indexes, which are just freed from the user and
     need to query the hardware counter to get the reset value before
     this counter can be reused again.

The counter pool also supports cache per HW steering's queues, which are
also based on the rte_ring of zero-copy variation.

The cache can be configured in size, preload, threshold, and fetch size,
they are all exposed via device args.

The main operations of the counter pool are as follows:

 - Get one counter from the pool:
   1. The user call _get_* API.
   2. If the cache is enabled, dequeue one counter index from the local
      cache:
      2. A: if the dequeued one from the local cache is still in reset
        status (counter's query_gen_when_free is equal to pool's query
        gen):
        I. Flush all counters in the local cache back to global
           wait_reset_list.
        II. Fetch _fetch_sz_ counters into the cache from the global
            free list.
        III. Fetch one counter from the cache.
   3. If the cache is empty, fetch _fetch_sz_ counters from the global
      free list into the cache and fetch one counter from the cache.
 - Free one counter into the pool:
   1. The user calls _put_* API.
   2. Put the counter into the local cache.
   3. If the local cache is full:
      A: Write back all counters above _threshold_ into the global
         wait_reset_list.
      B: Also, write back this counter into the global wait_reset_list.

When the local cache is disabled, _get_/_put_ cache directly from/into
global list.

Signed-off-by: Xiaoyu Min <jackmin@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:39 +02:00
Alexander Kozyrev
24865366e4 net/mlx5: support flow meter action for HWS
This commit adds meter action for HWS steering.

HW steering meter is based on ASO. The number of meters will
be used by flows should be specified in advance in the flow
configure API.

Signed-off-by: Alexander Kozyrev <akozyrev@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:39 +02:00
Bing Zhao
ddb68e4733 net/mlx5: add extended metadata mode for HWS
The new mode 4 of devarg "dv_xmeta_en" is added for HWS only. In this
mode, the Rx / Tx metadata with 32b width copy between FDB and NIC is
supported.

The mark is only supported in NIC and there is no copy supported.

Signed-off-by: Bing Zhao <bingz@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:38 +02:00
Dariusz Sosnowski
1939eb6f66 net/mlx5: support flow port action with HWS
This patch implements creating and caching of port action for use with
HW Steering FDB flows.

Actions are created on flow template API configuration and created
only on the port designated as the master. Attaching and detaching ports
in the same switching domain causes an update to the port actions cache
by, respectively, creating and destroying actions.

A new devarg fdb_def_rule_en is being added and it's used to control
the default dedicated E-Switch rules that are created by the PMD
implicitly or not, and PMD sets this value to 1 by default.

If set to 0, the default E-Switch rule will not be created and the user
can create the specific E-Switch rules on the root table if needed.

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:38 +02:00
Suanming Mou
0f4aa72b99 net/mlx5: support flow modify field with HWS
This patch introduces support for modify_field rte_flow actions in HWS
mode that includes:
	- Ingress and egress domains,
	- SET and ADD operations,
	- usage of arbitrary bit offsets and widths for packet and metadata
	  fields.

This is implemented in two phases:
1. On flow table creation the hardware commands are generated, based
   on rte_flow action templates, and stored alongside action template.

2. On flow rule creation/queueing the hardware commands are updated with
   values provided by the user. Any masks over immediate values, provided
   in action templates, are applied to these values before enqueueing rules
   for creation.

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:38 +02:00
Suanming Mou
7f6daa490d net/mlx5: add shared header reformat
As the rte_flow_async API defines the action mask with a field value
not being 0 means the action will be used as shared in all the flows
in the table.

The header reformat action with the action mask field not being 0 will
be created as constant shared action. For encapsulation header reformat
action, there are two kinds of encapsulation data, raw_encap_data
and rte_flow_item encap_data. Both of these two kinds of data can be
identified from the action mask conf as constant or not.

Examples:
1. VXLAN encap (encap_data: rte_flow_item)
	action conf (eth/ipv4/udp/vxlan_hdr)

	a. action mask conf (eth/ipv4/udp/vxlan_hdr)
	  - items are constant.
	b. action mask conf (NULL)
	  - items will change.

2. RAW encap (encap_data: raw)
	action conf (raw_data)

	a. action mask conf (not NULL)
	  - encap_data constant.
	b. action mask conf (NULL)
	  - encap_data will change.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:37 +02:00
Suanming Mou
b206c558f7 net/mlx5: fix IPv6 and TCP RSS hash fields
In the flow_dv_hashfields_set() function, while item_flags was 0,
the code went directly to the first if and the else case would
never have a chance to be checked. This caused the IPv6 and TCP hash
fields in the else case would never be set.

This commit adds the dedicated HW steering hash field set function
to generate the RSS hash fields.

Fixes: 3a2f674b6a ("net/mlx5: add queue and RSS HW steering action")
Cc: stable@dpdk.org

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:37 +02:00
Suanming Mou
4fb0ef2976 net/mlx5: fix steering engine type check
In the function flow_get_drv_type(), attr will be read in non-HWS mode.
In case the user calls the HWS API in SWS mode, the attr should be
placed in HWS functions or it will cause a crash.

Fixes: c40c061a02 ("net/mlx5: add basic flow queue operation")
Cc: stable@dpdk.org

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
2022-10-26 13:33:36 +02:00
Alex Vesker
22681deead net/mlx5/hws: enable hardware steering
Replace stub implementation of HWS with mlx5dr code.

Signed-off-by: Alex Vesker <valex@nvidia.com>
2022-10-26 13:33:36 +02:00
Hamdan Igbaria
78580cf4e7 net/mlx5/hws: add debug layer
The debug layer is used to generate a debug CSV file
containing details of the context, table, matcher, rules
and other useful debug information.

Signed-off-by: Hamdan Igbaria <hamdani@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
2022-10-26 13:33:36 +02:00
Erez Shitrit
f8c8a6d844 net/mlx5/hws: add action object
Action objects are used for executing different HW actions
over packets. Each action contains the HW resources and parameters
needed for action use over the HW when creating a rule.

Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
2022-10-26 13:33:35 +02:00
Alex Vesker
405242c52d net/mlx5/hws: add rule object
HWS rule objects reside under the matcher, each rule holds
the configuration for the packet fields to match and the
set of actions to execute over the packet that has the requested
fields.

Rules can be created asynchronously in parallel over multiple
queues to different matchers with each rule configured to the HW.

Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
2022-10-26 13:33:35 +02:00
Alex Vesker
c467608215 net/mlx5/hws: add matcher object
HWS matcher resides under the table object, each table can
have multiple chained matches with different attributes.

Each matcher represents a combination of match and action templates,
and can contain multiple configurations based on the templates.

Packets are steered from the table to the matcher and from there to
other objects.

The matcher allows efficient HW packet field matching and action
execution based on the configuration done to it.

Signed-off-by: Alex Vesker <valex@nvidia.com>
Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
2022-10-26 13:33:34 +02:00
Alex Vesker
394cc7ba40 net/mlx5/hws: add table object
HWS table resides under the context object, each context can
have multiple tables with different steering types RX/TX/FDB.

The table is not only a logical object but it is also represented
in the HW, packets can be steered to the table, and from there
to other tables.

Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
2022-10-26 13:33:34 +02:00
Alex Vesker
b0290e56dd net/mlx5/hws: add context object
Context is the first mlx5dr object created, all sub objects
table, matcher, rule, and action are created using the context.
The context holds the capabilities and the send queues used for
configuring the offloads to the HW.

Signed-off-by: Alex Vesker <valex@nvidia.com>
2022-10-26 13:33:34 +02:00
Alex Vesker
c55c2bf353 net/mlx5/hws: add definer layer
Definers are HW objects that are used for matching, rte items
are translated to definers, each definer holds the fields and
bit-masks used for HW flow matching. The definer layer is used
for finding the most efficient definer for each set of items.
In addition to definer creation we also calculate the field
copy (fc) array used for efficient items to WQE conversion.

Signed-off-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
2022-10-26 13:33:33 +02:00
Alex Vesker
3eb748869d net/mlx5/hws: add send layer
HWS configures flows to the HW using a QP, each WQE has
the details of the flow we want to offload. The send layer
allocates the resources needed to send the request to the HW
as well as managing the queues, getting completions and
handling failures.

Signed-off-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
2022-10-26 13:33:33 +02:00
Erez Shitrit
b4dd7bcb0d net/mlx5/hws: add pool and buddy
HWS needs to manage different types of device memory in
an efficient and quick way. For this, memory pools are
being used.

Signed-off-by: Erez Shitrit <erezsh@nvidia.com>
Signed-off-by: Alex Vesker <valex@nvidia.com>
2022-10-26 13:33:33 +02:00