d/numam-dpdk
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

ethdev: add proactive error handling mode

Browse Source 
Some PMDs (e.g. hns3) could detect hardware or firmware errors, one
error recovery mode is to report RTE_ETH_EVENT_INTR_RESET event, and
wait for application invoke rte_eth_dev_reset() to recover the port,
however, this mode has the following weaknesses:

1) Due to different hardware and software design, some NIC port recovery
process requires multiple handshakes with the firmware and PF (when the
port is VF). It takes a long time to complete the entire operation for
one port, If multiple ports (for example, multiple VFs of a PF) are
reset at the same time, other VFs may fail to be reset. (Because the
reset processing is serial, the previous VFs must be processed before
the subsequent VFs).

2) The impact on the application layer is great, and it should stop
working queues, stop calling Rx and Tx functions, and then call
rte_eth_dev_reset(), and re-setup all again.

This patch introduces proactive error handling mode, the PMD will try
to recover from the errors itself. In this process, the PMD sets the
data path pointers to dummy functions (which will prevent the crash),
and also make sure the control path operations failed with retcode
-EBUSY.

Because the PMD recovers automatically, the application can only sense
that the data flow is disconnected for a while and the control API
returns an error in this period.

In order to sense the error happening/recovering, three events were
introduced:

1) RTE_ETH_EVENT_ERR_RECOVERING: used to notify the application that it
detected an error and the recovery is being started. Upon receiving the
event, the application should not invoke any control path APIs until
receiving RTE_ETH_EVENT_RECOVERY_SUCCESS or
RTE_ETH_EVENT_RECOVERY_FAILED event.

2) RTE_ETH_EVENT_RECOVERY_SUCCESS: used to notify the application that
it recovers successful from the error, the PMD already re-configures the
port, and the effect is the same as that of the restart operation.

3) RTE_ETH_EVENT_RECOVERY_FAILED: used to notify the application that it
recovers failed from the error, the port should not usable anymore. The
application should close the port.

Signed-off-by: Kalesh AP <kalesh-anakkur.purayil@broadcom.com>
Signed-off-by: Somnath Kotur <somnath.kotur@broadcom.com>
Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
Reviewed-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
This commit is contained in:
Kalesh AP 2022-10-13 12:42:24 +00:00 committed by Andrew Rybchenko
parent 0d5c38bac7
commit eb0d471a89
7 changed files with 135 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
app/test-pmd/config.c
View File

@ -929,6 +929,9 @@ port_infos_display(portid_t port_id)
case RTE_ETH_ERROR_HANDLE_MODE_PASSIVE:
printf("passive\n");
break;
case RTE_ETH_ERROR_HANDLE_MODE_PROACTIVE:
printf("proactive\n");
break;
default:
printf("unknown\n");
break;

10
app/test-pmd/parameters.c
View File

@ -167,9 +167,9 @@ usage(char* progname)
printf(" --no-rmv-interrupt: disable device removal interrupt.\n");
printf(" --bitrate-stats=N: set the logical core N to perform "
"bit-rate calculation.\n");
printf(" --print-event <unknown|intr_lsc|queue_state|intr_reset|vf_mbox|macsec|intr_rmv|flow_aged|all>: "
printf(" --print-event <unknown|intr_lsc|queue_state|intr_reset|vf_mbox|macsec|intr_rmv|flow_aged|err_recovering|recovery_success|recovery_failed|all>: "
"enable print of designated event or all of them.\n");
printf(" --mask-event <unknown|intr_lsc|queue_state|intr_reset|vf_mbox|macsec|intr_rmv|flow_aged|all>: "
printf(" --mask-event <unknown|intr_lsc|queue_state|intr_reset|vf_mbox|macsec|intr_rmv|flow_aged|err_recovering|recovery_success|recovery_failed||all>: "
"disable print of designated event or all of them.\n");
printf(" --flow-isolate-all: "
"requests flow API isolated mode on all ports at initialization time.\n");
@ -453,6 +453,12 @@ parse_event_printing_config(const char *optarg, int enable)
mask = UINT32_C(1) << RTE_ETH_EVENT_DESTROY;
else if (!strcmp(optarg, "flow_aged"))
mask = UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED;
else if (!strcmp(optarg, "err_recovering"))
mask = UINT32_C(1) << RTE_ETH_EVENT_ERR_RECOVERING;
else if (!strcmp(optarg, "recovery_success"))
mask = UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_SUCCESS;
else if (!strcmp(optarg, "recovery_failed"))
mask = UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_FAILED;
else if (!strcmp(optarg, "all"))
mask = ~UINT32_C(0);
else {

8
app/test-pmd/testpmd.c
View File

@ -426,6 +426,9 @@ static const char * const eth_event_desc[] = {
[RTE_ETH_EVENT_DESTROY] = "device released",
[RTE_ETH_EVENT_FLOW_AGED] = "flow aged",
[RTE_ETH_EVENT_RX_AVAIL_THRESH] = "RxQ available descriptors threshold reached",
[RTE_ETH_EVENT_ERR_RECOVERING] = "error recovering",
[RTE_ETH_EVENT_RECOVERY_SUCCESS] = "error recovery successful",
[RTE_ETH_EVENT_RECOVERY_FAILED] = "error recovery failed",
[RTE_ETH_EVENT_MAX] = NULL,
};
@ -440,7 +443,10 @@ uint32_t event_print_mask = (UINT32_C(1) << RTE_ETH_EVENT_UNKNOWN) |
(UINT32_C(1) << RTE_ETH_EVENT_IPSEC) |
(UINT32_C(1) << RTE_ETH_EVENT_MACSEC) |
(UINT32_C(1) << RTE_ETH_EVENT_INTR_RMV) |
(UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED);
(UINT32_C(1) << RTE_ETH_EVENT_FLOW_AGED) |
(UINT32_C(1) << RTE_ETH_EVENT_ERR_RECOVERING) |
(UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_SUCCESS) |
(UINT32_C(1) << RTE_ETH_EVENT_RECOVERY_FAILED);
/*
* Decide if all memory are locked for performance.
*/

49
doc/guides/prog_guide/poll_mode_drv.rst
View File

@ -627,3 +627,52 @@ by application.
The PMD itself should not call rte_eth_dev_reset(). The PMD can trigger
the application to handle reset event. It is duty of application to
handle all synchronization before it calls rte_eth_dev_reset().
The above error handling mode is known as ``RTE_ETH_ERROR_HANDLE_MODE_PASSIVE``.
Proactive Error Handling Mode
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This mode is known as ``RTE_ETH_ERROR_HANDLE_MODE_PROACTIVE``,
different from the application invokes recovery in PASSIVE mode,
the PMD automatically recovers from error in PROACTIVE mode,
and only a small amount of work is required for the application.
During error detection and automatic recovery,
the PMD sets the data path pointers to dummy functions
(which will prevent the crash),
and also make sure the control path operations fail with a return code ``-EBUSY``.
Because the PMD recovers automatically,
the application can only sense that the data flow is disconnected for a while
and the control API returns an error in this period.
In order to sense the error happening/recovering,
as well as to restore some additional configuration,
three events are available:
``RTE_ETH_EVENT_ERR_RECOVERING``
Notify the application that an error is detected
and the recovery is being started.
Upon receiving the event, the application should not invoke
any control path function until receiving
``RTE_ETH_EVENT_RECOVERY_SUCCESS`` or ``RTE_ETH_EVENT_RECOVERY_FAILED`` event.
.. note::
Before the PMD reports the recovery result,
the PMD may report the ``RTE_ETH_EVENT_ERR_RECOVERING`` event again,
because a larger error may occur during the recovery.
``RTE_ETH_EVENT_RECOVERY_SUCCESS``
Notify the application that the recovery from error is successful,
the PMD already re-configures the port,
and the effect is the same as a restart operation.
``RTE_ETH_EVENT_RECOVERY_FAILED``
Notify the application that the recovery from error failed,
the port should not be usable anymore.
The application should close the port.
The error handling mode supported by the PMD can be reported through
``rte_eth_dev_info_get``.

7
doc/guides/rel_notes/release_22_11.rst
View File

@ -82,6 +82,13 @@ New Features
* Supported protocol-based buffer split using added ``proto_hdr``
in structure ``rte_eth_rxseg_split``.
* **Added proactive error handling mode for ethdev.**
Added proactive error handling mode for ethdev,
and three events were introduced: ``RTE_ETH_EVENT_ERR_RECOVERING``
to report that the port is recovering from an error,
``RTE_ETH_EVENT_RECOVER_SUCCESS`` and ``RTE_ETH_EVENT_RECOVER_FAILED``.
* **Added ethdev Rx/Tx descriptor dump API.**
Added the ethdev Rx/Tx descriptor dump API which provides functions

4
doc/guides/testpmd_app_ug/run_app.rst
View File

@ -414,12 +414,12 @@ The command line options are:
Set the logical core N to perform bitrate calculation.
* ``--print-event <unknown|intr_lsc|queue_state|intr_reset|vf_mbox|macsec|intr_rmv|dev_probed|dev_released|flow_aged|all>``
* ``--print-event <unknown|intr_lsc|queue_state|intr_reset|vf_mbox|macsec|intr_rmv|dev_probed|dev_released|flow_aged|err_recovering|recovery_success|recovery_failed|all>``
Enable printing the occurrence of the designated event. Using all will
enable all of them.
* ``--mask-event <unknown|intr_lsc|queue_state|intr_reset|vf_mbox|macsec|intr_rmv|dev_probed|dev_released|flow_aged|all>``
* ``--mask-event <unknown|intr_lsc|queue_state|intr_reset|vf_mbox|macsec|intr_rmv|dev_probed|dev_released|flow_aged|err_recovering|recovery_success|recovery_failed|all>``
Disable printing the occurrence of the designated event. Using all will
disable all of them.

59
lib/ethdev/rte_ethdev.h
View File

@ -1700,6 +1700,12 @@ enum rte_eth_err_handle_mode {
* and the application invokes @see rte_eth_dev_reset to recover the port.
*/
RTE_ETH_ERROR_HANDLE_MODE_PASSIVE,
/** Proactive error handling, after the PMD detects that a reset is required,
* the PMD reports @see RTE_ETH_EVENT_ERR_RECOVERING event,
* do recovery internally, and finally reports the recovery result event
* (@see RTE_ETH_EVENT_RECOVERY_*).
*/
RTE_ETH_ERROR_HANDLE_MODE_PROACTIVE,
};
/**
@ -3886,6 +3892,59 @@ enum rte_eth_event_type {
* @see rte_eth_rx_avail_thresh_set()
*/
RTE_ETH_EVENT_RX_AVAIL_THRESH,
/** Port recovering from a hardware or firmware error.
* If PMD supports proactive error recovery,
* it should trigger this event to notify application
* that it detected an error and the recovery is being started.
* Upon receiving the event, the application should not invoke any control path API
* (such as rte_eth_dev_configure/rte_eth_dev_stop...) until receiving
* RTE_ETH_EVENT_RECOVERY_SUCCESS or RTE_ETH_EVENT_RECOVERY_FAILED event.
* The PMD will set the data path pointers to dummy functions,
* and re-set the data path pointers to non-dummy functions
* before reporting RTE_ETH_EVENT_RECOVERY_SUCCESS event.
* It means that the application cannot send or receive any packets
* during this period.
* @note Before the PMD reports the recovery result,
* the PMD may report the RTE_ETH_EVENT_ERR_RECOVERING event again,
* because a larger error may occur during the recovery.
*/
RTE_ETH_EVENT_ERR_RECOVERING,
/** Port recovers successfully from the error.
* The PMD already re-configured the port,
* and the effect is the same as a restart operation.
* a) The following operation will be retained: (alphabetically)
* - DCB configuration
* - FEC configuration
* - Flow control configuration
* - LRO configuration
* - LSC configuration
* - MTU
* - MAC address (default and those supplied by MAC address array)
* - Promiscuous and allmulticast mode
* - PTP configuration
* - Queue (Rx/Tx) settings
* - Queue statistics mappings
* - RSS configuration by rte_eth_dev_rss_xxx() family
* - Rx checksum configuration
* - Rx interrupt settings
* - Traffic management configuration
* - VLAN configuration (including filtering, tpid, strip, pvid)
* - VMDq configuration
* b) The following configuration maybe retained
* or not depending on the device capabilities:
* - flow rules
* (@see RTE_ETH_DEV_CAPA_FLOW_RULE_KEEP)
* - shared flow objects
* (@see RTE_ETH_DEV_CAPA_FLOW_SHARED_OBJECT_KEEP)
* c) Any other configuration will not be stored
* and will need to be re-configured.
*/
RTE_ETH_EVENT_RECOVERY_SUCCESS,
/** Port recovery failed.
* It means that the port should not be usable anymore.
* The application should close the port.
*/
RTE_ETH_EVENT_RECOVERY_FAILED,
RTE_ETH_EVENT_MAX /**< max value of this enum */
};