d/freebsd-skq
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
c19ed2c875
freebsd-skq/sys/dev/ena/ena.c

3595 lines
96 KiB
C
Raw Normal View History

Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/*-
* BSD LICENSE
*
Update ENA version to v2.0.0 ENAv2 introduces many new features, bug fixes and improvements. Main new features are LLQ (Low Latency Queues) and independent queues reconfiguration using sysctl commands. The year in copyright notice was updated to 2019. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:52:32 +00:00
* Copyright (c) 2015-2019 Amazon.com, Inc. or its affiliates.
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/bus.h>
#include <sys/endian.h>
#include <sys/kernel.h>
#include <sys/kthread.h>
#include <sys/malloc.h>
#include <sys/mbuf.h>
#include <sys/module.h>
#include <sys/rman.h>
#include <sys/smp.h>
#include <sys/socket.h>
#include <sys/sockio.h>
#include <sys/sysctl.h>
#include <sys/taskqueue.h>
#include <sys/time.h>
#include <sys/eventhandler.h>
#include <machine/bus.h>
#include <machine/resource.h>
#include <machine/in_cksum.h>
#include <net/bpf.h>
#include <net/ethernet.h>
#include <net/if.h>
#include <net/if_var.h>
#include <net/if_arp.h>
#include <net/if_dl.h>
#include <net/if_media.h>
#include <net/if_types.h>
#include <net/if_vlan_var.h>
#include <netinet/in_systm.h>
#include <netinet/in.h>
#include <netinet/if_ether.h>
#include <netinet/ip.h>
#include <netinet/ip6.h>
#include <netinet/tcp.h>
#include <netinet/udp.h>
#include <dev/pci/pcivar.h>
#include <dev/pci/pcireg.h>
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
#include <vm/vm.h>
#include <vm/pmap.h>
Split Rx/Tx from initialization code in ENA driver Move Rx/Tx routines to separate file. Some functions: * ena_restore_device, * ena_destroy_device, * ena_up, * ena_down, * ena_refill_rx_bufs could be reused in upcoming netmap code in the driver. To make it possible, they were moved to ena.h header. Differential Revision: https://reviews.freebsd.org/D21933 Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:44:26 +00:00
#include "ena_datapath.h"
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
#include "ena.h"
#include "ena_sysctl.h"
Introduce NETMAP support in ENA Mock implementation of NETMAP routines is located in ena_netmap.c/.h files. All code is protected under the DEV_NETMAP macro. Makefile was updated with files and flag. As ENA driver provide own implementations of (un)likely it must be undefined before including NETMAP headers. ena_netmap_attach function is called on the end of NIC attach. It fills structure with NIC configuration and callbacks. Then provides it to netmap_attach. Similarly netmap_detach is called during ena_detach. Three callbacks are used. nm_register is implemented by ena_netmap_reg. It is called when user space application open or close NIC in NETMAP mode. Current action is recognized based on onoff parameter: true means on and false off. As NICs rings need to be reconfigured ena_down and ena_up are reused. When user space application wants to receive new packets from NIC nm_rxsync is called, and when there are new packets ready for Tx nm_txsync is called. Differential Revision: https://reviews.freebsd.org/D21934 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:51:18 +00:00
#ifdef DEV_NETMAP
#include "ena_netmap.h"
#endif /* DEV_NETMAP */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/*********************************************************
* Function prototypes
*********************************************************/
static int ena_probe(device_t);
static void ena_intr_msix_mgmnt(void *);
static void ena_free_pci_resources(struct ena_adapter *);
static int ena_change_mtu(if_t, int);
static inline void ena_alloc_counters(counter_u64_t *, int);
static inline void ena_free_counters(counter_u64_t *, int);
static inline void ena_reset_counters(counter_u64_t *, int);
static void ena_init_io_rings_common(struct ena_adapter *,
struct ena_ring *, uint16_t);
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
static void ena_init_io_rings(struct ena_adapter *);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
static void ena_free_io_ring_resources(struct ena_adapter *, unsigned int);
static void ena_free_all_io_rings_resources(struct ena_adapter *);
static int ena_setup_tx_dma_tag(struct ena_adapter *);
static int ena_free_tx_dma_tag(struct ena_adapter *);
static int ena_setup_rx_dma_tag(struct ena_adapter *);
static int ena_free_rx_dma_tag(struct ena_adapter *);
Add support for ENA NETMAP Tx Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:59:29 +00:00
static void ena_release_all_tx_dmamap(struct ena_ring *);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
static int ena_setup_tx_resources(struct ena_adapter *, int);
static void ena_free_tx_resources(struct ena_adapter *, int);
static int ena_setup_all_tx_resources(struct ena_adapter *);
static void ena_free_all_tx_resources(struct ena_adapter *);
static int ena_setup_rx_resources(struct ena_adapter *, unsigned int);
static void ena_free_rx_resources(struct ena_adapter *, unsigned int);
static int ena_setup_all_rx_resources(struct ena_adapter *);
static void ena_free_all_rx_resources(struct ena_adapter *);
static inline int ena_alloc_rx_mbuf(struct ena_adapter *, struct ena_ring *,
struct ena_rx_buffer *);
static void ena_free_rx_mbuf(struct ena_adapter *, struct ena_ring *,
struct ena_rx_buffer *);
static void ena_free_rx_bufs(struct ena_adapter *, unsigned int);
static void ena_refill_all_rx_bufs(struct ena_adapter *);
static void ena_free_all_rx_bufs(struct ena_adapter *);
static void ena_free_tx_bufs(struct ena_adapter *, unsigned int);
static void ena_free_all_tx_bufs(struct ena_adapter *);
static void ena_destroy_all_tx_queues(struct ena_adapter *);
static void ena_destroy_all_rx_queues(struct ena_adapter *);
static void ena_destroy_all_io_queues(struct ena_adapter *);
static int ena_create_io_queues(struct ena_adapter *);
Lock optimization in ENA Handle IO interrupts using filter routine. That way, the main cleanup task could be moved to the separate thread using taskqueue. The deferred Rx cleanup task was removed, and now the cleanup task is begin called instead. That way, the Rx lock could be removed. In addition, Queue management (wake up and stop TX ring) was added, so the TX cleanup task can be performed mostly lockless. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:29:24 +00:00
static int ena_handle_msix(void *);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
static int ena_enable_msix(struct ena_adapter *);
static void ena_setup_mgmnt_intr(struct ena_adapter *);
Fix NULL pointer dereference in ena_up() If the call to ena_up() in ena_restore_device() fails, next usage of `ifconfig up` will cause NULL pointer dereference. This patch adds additional checks to prevent that. Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:42:52 +00:00
static int ena_setup_io_intr(struct ena_adapter *);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
static int ena_request_mgmnt_irq(struct ena_adapter *);
static int ena_request_io_irq(struct ena_adapter *);
static void ena_free_mgmnt_irq(struct ena_adapter *);
static void ena_free_io_irq(struct ena_adapter *);
static void ena_free_irqs(struct ena_adapter*);
static void ena_disable_msix(struct ena_adapter *);
static void ena_unmask_all_io_irqs(struct ena_adapter *);
static int ena_rss_configure(struct ena_adapter *);
static int ena_up_complete(struct ena_adapter *);
static uint64_t ena_get_counter(if_t, ift_counter);
static int ena_media_change(if_t);
static void ena_media_status(if_t, struct ifmediareq *);
static void ena_init(void *);
static int ena_ioctl(if_t, u_long, caddr_t);
static int ena_get_dev_offloads(struct ena_com_dev_get_features_ctx *);
static void ena_update_host_info(struct ena_admin_host_info *, if_t);
static void ena_update_hwassist(struct ena_adapter *);
static int ena_setup_ifnet(device_t, struct ena_adapter *,
struct ena_com_dev_get_features_ctx *);
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
static int ena_enable_wc(struct resource *);
static int ena_set_queues_placement_policy(device_t, struct ena_com_dev *,
struct ena_admin_feature_llq_desc *, struct ena_llq_configurations *);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
static int ena_calc_io_queue_num(struct ena_adapter *,
struct ena_com_dev_get_features_ctx *);
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
static int ena_calc_queue_size(struct ena_adapter *,
struct ena_calc_queue_size_ctx *);
static int ena_handle_updated_queues(struct ena_adapter *,
struct ena_com_dev_get_features_ctx *);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
static int ena_rss_init_default(struct ena_adapter *);
static void ena_rss_init_default_deferred(void *);
Fill bdf field of the host_info structure in ENA The host info bdf field is the abbreviation for the bus, device, function of the PCI on which the device is being attached to. Now the driver is filling information about that using FreeBSD RID resource. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:35:02 +00:00
static void ena_config_host_info(struct ena_com_dev *, device_t);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
static int ena_attach(device_t);
static int ena_detach(device_t);
static int ena_device_init(struct ena_adapter *, device_t,
struct ena_com_dev_get_features_ctx *, int *);
static int ena_enable_msix_and_set_admin_interrupts(struct ena_adapter *,
int);
static void ena_update_on_link_change(void *, struct ena_admin_aenq_entry *);
static void unimplemented_aenq_handler(void *,
struct ena_admin_aenq_entry *);
static void ena_timer_service(void *);
static char ena_version[] = DEVICE_NAME DRV_MODULE_NAME " v" DRV_MODULE_VERSION;
static ena_vendor_info_t ena_vendor_info_array[] = {
{ PCI_VENDOR_ID_AMAZON, PCI_DEV_ID_ENA_PF, 0},
{ PCI_VENDOR_ID_AMAZON, PCI_DEV_ID_ENA_LLQ_PF, 0},
{ PCI_VENDOR_ID_AMAZON, PCI_DEV_ID_ENA_VF, 0},
{ PCI_VENDOR_ID_AMAZON, PCI_DEV_ID_ENA_LLQ_VF, 0},
/* Last entry */
{ 0, 0, 0 }
};
/*
* Contains pointers to event handlers, e.g. link state chage.
*/
static struct ena_aenq_handlers aenq_handlers;
void
ena_dmamap_callback(void *arg, bus_dma_segment_t *segs, int nseg, int error)
{
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (error != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return;
*(bus_addr_t *) arg = segs[0].ds_addr;
}
int
ena_dma_alloc(device_t dmadev, bus_size_t size,
ena_mem_handle_t *dma , int mapflags)
{
struct ena_adapter* adapter = device_get_softc(dmadev);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
uint32_t maxsize;
uint64_t dma_space_addr;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
int error;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
maxsize = ((size - 1) / PAGE_SIZE + 1) * PAGE_SIZE;
dma_space_addr = ENA_DMA_BIT_MASK(adapter->dma_width);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(dma_space_addr == 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
dma_space_addr = BUS_SPACE_MAXADDR;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
error = bus_dma_tag_create(bus_get_dma_tag(dmadev), /* parent */
Fix creation of dma tags and TSO settings TSO settings were not reflecting real HW capabilities. DMA tags were created with wrong window - high address was the same as low, so excluding window was not working. Capabilities of TX dma transaction were not set properly - TSO max size had been increased and size of one segment had been adjusted. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-04 00:08:47 +00:00
8, 0, /* alignment, bounds */
dma_space_addr, /* lowaddr of exclusion window */
BUS_SPACE_MAXADDR,/* highaddr of exclusion window */
NULL, NULL, /* filter, filterarg */
maxsize, /* maxsize */
1, /* nsegments */
maxsize, /* maxsegsize */
BUS_DMA_ALLOCNOW, /* flags */
NULL, /* lockfunc */
NULL, /* lockarg */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
&dma->tag);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(error != 0)) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_ALERT, "bus_dma_tag_create failed: %d\n", error);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
goto fail_tag;
}
error = bus_dmamem_alloc(dma->tag, (void**) &dma->vaddr,
BUS_DMA_COHERENT | BUS_DMA_ZERO, &dma->map);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(error != 0)) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_ALERT, "bus_dmamem_alloc(%ju) failed: %d\n",
(uintmax_t)size, error);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
goto fail_map_create;
}
dma->paddr = 0;
error = bus_dmamap_load(dma->tag, dma->map, dma->vaddr,
size, ena_dmamap_callback, &dma->paddr, mapflags);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely((error != 0) || (dma->paddr == 0))) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_ALERT, ": bus_dmamap_load failed: %d\n", error);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
goto fail_map_load;
}
Fix DMA synchronization in the ENA driver Tx and Rx paths The DMA in FreeBSD requires explicit synchronization. ENA driver was only doing PREREAD and PREWRITE synchronizations. Missing bus_dmamap_sync() calls were added. It is also required to synchronize DMA engine before unloading DMA map. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:18:23 +00:00
bus_dmamap_sync(dma->tag, dma->map,
BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (0);
fail_map_load:
bus_dmamem_free(dma->tag, dma->vaddr, dma->map);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
fail_map_create:
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
bus_dma_tag_destroy(dma->tag);
fail_tag:
dma->tag = NULL;
Set vaddr and paddr as NULL when DMA alloc fails in ENA To prevent errors from assigning values from the DMA structure in case of an error, zero the vaddr and paddr values upon failure. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:19:32 +00:00
dma->vaddr = NULL;
dma->paddr = 0;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (error);
}
static void
ena_free_pci_resources(struct ena_adapter *adapter)
{
device_t pdev = adapter->pdev;
if (adapter->memory != NULL) {
bus_release_resource(pdev, SYS_RES_MEMORY,
PCIR_BAR(ENA_MEM_BAR), adapter->memory);
}
if (adapter->registers != NULL) {
bus_release_resource(pdev, SYS_RES_MEMORY,
PCIR_BAR(ENA_REG_BAR), adapter->registers);
}
}
static int
ena_probe(device_t dev)
{
ena_vendor_info_t *ent;
char adapter_name[60];
uint16_t pci_vendor_id = 0;
uint16_t pci_device_id = 0;
pci_vendor_id = pci_get_vendor(dev);
pci_device_id = pci_get_device(dev);
ent = ena_vendor_info_array;
while (ent->vendor_id != 0) {
if ((pci_vendor_id == ent->vendor_id) &&
(pci_device_id == ent->device_id)) {
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
ena_trace(ENA_DBG, "vendor=%x device=%x\n",
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
pci_vendor_id, pci_device_id);
sprintf(adapter_name, DEVICE_DESC);
device_set_desc_copy(dev, adapter_name);
return (BUS_PROBE_DEFAULT);
}
ent++;
}
return (ENXIO);
}
static int
ena_change_mtu(if_t ifp, int new_mtu)
{
struct ena_adapter *adapter = if_getsoftc(ifp);
Read max MTU from the ENA device The device now provides driver with max available MTU value it can handle. The function setting MTU for the interface was simplified and reworked to follow up this changes. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12870
2017-11-09 13:35:07 +00:00
int rc;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Read max MTU from the ENA device The device now provides driver with max available MTU value it can handle. The function setting MTU for the interface was simplified and reworked to follow up this changes. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12870
2017-11-09 13:35:07 +00:00
if ((new_mtu > adapter->max_mtu) || (new_mtu < ENA_MIN_MTU)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "Invalid MTU setting. "
Read max MTU from the ENA device The device now provides driver with max available MTU value it can handle. The function setting MTU for the interface was simplified and reworked to follow up this changes. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12870
2017-11-09 13:35:07 +00:00
"new_mtu: %d max mtu: %d min mtu: %d\n",
new_mtu, adapter->max_mtu, ENA_MIN_MTU);
return (EINVAL);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
rc = ena_com_set_dev_mtu(adapter->ena_dev, new_mtu);
Read max MTU from the ENA device The device now provides driver with max available MTU value it can handle. The function setting MTU for the interface was simplified and reworked to follow up this changes. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12870
2017-11-09 13:35:07 +00:00
if (likely(rc == 0)) {
ena_trace(ENA_DBG, "set MTU to %d\n", new_mtu);
if_setmtu(ifp, new_mtu);
} else {
device_printf(adapter->pdev, "Failed to set MTU to %d\n",
new_mtu);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Read max MTU from the ENA device The device now provides driver with max available MTU value it can handle. The function setting MTU for the interface was simplified and reworked to follow up this changes. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12870
2017-11-09 13:35:07 +00:00
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
static inline void
ena_alloc_counters(counter_u64_t *begin, int size)
{
counter_u64_t *end = (counter_u64_t *)((char *)begin + size);
for (; begin < end; ++begin)
*begin = counter_u64_alloc(M_WAITOK);
}
static inline void
ena_free_counters(counter_u64_t *begin, int size)
{
counter_u64_t *end = (counter_u64_t *)((char *)begin + size);
for (; begin < end; ++begin)
counter_u64_free(*begin);
}
static inline void
ena_reset_counters(counter_u64_t *begin, int size)
{
counter_u64_t *end = (counter_u64_t *)((char *)begin + size);
for (; begin < end; ++begin)
counter_u64_zero(*begin);
}
static void
ena_init_io_rings_common(struct ena_adapter *adapter, struct ena_ring *ring,
uint16_t qid)
{
ring->qid = qid;
ring->adapter = adapter;
ring->ena_dev = adapter->ena_dev;
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
ring->first_interrupt = false;
ring->no_interrupt_event_cnt = 0;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
static void
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_init_io_rings(struct ena_adapter *adapter)
{
struct ena_com_dev *ena_dev;
struct ena_ring *txr, *rxr;
struct ena_que *que;
int i;
ena_dev = adapter->ena_dev;
for (i = 0; i < adapter->num_queues; i++) {
txr = &adapter->tx_ring[i];
rxr = &adapter->rx_ring[i];
/* TX/RX common ring state */
ena_init_io_rings_common(adapter, txr, i);
ena_init_io_rings_common(adapter, rxr, i);
/* TX specific ring state */
txr->ring_size = adapter->tx_ring_size;
txr->tx_max_header_size = ena_dev->tx_max_header_size;
txr->tx_mem_queue_type = ena_dev->tx_mem_queue_type;
txr->smoothed_interval =
ena_com_get_nonadaptive_moderation_interval_tx(ena_dev);
/* Allocate a buf ring */
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
txr->buf_ring_size = adapter->buf_ring_size;
txr->br = buf_ring_alloc(txr->buf_ring_size, M_DEVBUF,
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
M_WAITOK, &txr->ring_mtx);
/* Alloc TX statistics. */
ena_alloc_counters((counter_u64_t *)&txr->tx_stats,
sizeof(txr->tx_stats));
/* RX specific ring state */
rxr->ring_size = adapter->rx_ring_size;
rxr->smoothed_interval =
ena_com_get_nonadaptive_moderation_interval_rx(ena_dev);
/* Alloc RX statistics. */
ena_alloc_counters((counter_u64_t *)&rxr->rx_stats,
sizeof(rxr->rx_stats));
/* Initialize locks */
snprintf(txr->mtx_name, nitems(txr->mtx_name), "%s:tx(%d)",
device_get_nameunit(adapter->pdev), i);
snprintf(rxr->mtx_name, nitems(rxr->mtx_name), "%s:rx(%d)",
device_get_nameunit(adapter->pdev), i);
mtx_init(&txr->ring_mtx, txr->mtx_name, NULL, MTX_DEF);
que = &adapter->que[i];
que->adapter = adapter;
que->id = i;
que->tx_ring = txr;
que->rx_ring = rxr;
txr->que = que;
rxr->que = que;
Check for Rx ring state to prevent from stall in the ENA driver In case when Rx ring is full and driver will fail to allocate Rx mbufs, the ring could be stalled. Keep alive is checking every second for Rx ring state, and if it is full for two cycles, then trigger rx_cleanup routine in another thread. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12856
2017-11-09 11:48:22 +00:00
rxr->empty_rx_queue = 0;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
static void
ena_free_io_ring_resources(struct ena_adapter *adapter, unsigned int qid)
{
struct ena_ring *txr = &adapter->tx_ring[qid];
struct ena_ring *rxr = &adapter->rx_ring[qid];
ena_free_counters((counter_u64_t *)&txr->tx_stats,
sizeof(txr->tx_stats));
ena_free_counters((counter_u64_t *)&rxr->rx_stats,
sizeof(rxr->rx_stats));
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
ENA_RING_MTX_LOCK(txr);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
drbr_free(txr->br, M_DEVBUF);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
ENA_RING_MTX_UNLOCK(txr);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
mtx_destroy(&txr->ring_mtx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
static void
ena_free_all_io_rings_resources(struct ena_adapter *adapter)
{
int i;
for (i = 0; i < adapter->num_queues; i++)
ena_free_io_ring_resources(adapter, i);
}
static int
ena_setup_tx_dma_tag(struct ena_adapter *adapter)
{
int ret;
/* Create DMA tag for Tx buffers */
ret = bus_dma_tag_create(bus_get_dma_tag(adapter->pdev),
Fix creation of dma tags and TSO settings TSO settings were not reflecting real HW capabilities. DMA tags were created with wrong window - high address was the same as low, so excluding window was not working. Capabilities of TX dma transaction were not set properly - TSO max size had been increased and size of one segment had been adjusted. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-04 00:08:47 +00:00
1, 0, /* alignment, bounds */
ENA_DMA_BIT_MASK(adapter->dma_width), /* lowaddr of excl window */
BUS_SPACE_MAXADDR, /* highaddr of excl window */
NULL, NULL, /* filter, filterarg */
ENA_TSO_MAXSIZE, /* maxsize */
adapter->max_tx_sgl_size - 1, /* nsegments */
ENA_TSO_MAXSIZE, /* maxsegsize */
0, /* flags */
NULL, /* lockfunc */
NULL, /* lockfuncarg */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
&adapter->tx_buf_tag);
return (ret);
}
static int
ena_free_tx_dma_tag(struct ena_adapter *adapter)
{
int ret;
ret = bus_dma_tag_destroy(adapter->tx_buf_tag);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (likely(ret == 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
adapter->tx_buf_tag = NULL;
return (ret);
}
static int
ena_setup_rx_dma_tag(struct ena_adapter *adapter)
{
int ret;
/* Create DMA tag for Rx buffers*/
Fix creation of dma tags and TSO settings TSO settings were not reflecting real HW capabilities. DMA tags were created with wrong window - high address was the same as low, so excluding window was not working. Capabilities of TX dma transaction were not set properly - TSO max size had been increased and size of one segment had been adjusted. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-04 00:08:47 +00:00
ret = bus_dma_tag_create(bus_get_dma_tag(adapter->pdev), /* parent */
1, 0, /* alignment, bounds */
ENA_DMA_BIT_MASK(adapter->dma_width), /* lowaddr of excl window */
BUS_SPACE_MAXADDR, /* highaddr of excl window */
NULL, NULL, /* filter, filterarg */
MJUM16BYTES, /* maxsize */
Allow usage of more RX descriptors than 1 in ENA driver Using only 1 descriptor on RX could be an issue, if system would be low on resources and could not provide driver with large chunks of contiguous memory. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12871
2017-11-09 13:36:42 +00:00
adapter->max_rx_sgl_size, /* nsegments */
Fix creation of dma tags and TSO settings TSO settings were not reflecting real HW capabilities. DMA tags were created with wrong window - high address was the same as low, so excluding window was not working. Capabilities of TX dma transaction were not set properly - TSO max size had been increased and size of one segment had been adjusted. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-04 00:08:47 +00:00
MJUM16BYTES, /* maxsegsize */
0, /* flags */
NULL, /* lockfunc */
NULL, /* lockarg */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
&adapter->rx_buf_tag);
return (ret);
}
static int
ena_free_rx_dma_tag(struct ena_adapter *adapter)
{
int ret;
ret = bus_dma_tag_destroy(adapter->rx_buf_tag);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (likely(ret == 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
adapter->rx_buf_tag = NULL;
return (ret);
}
Add support for ENA NETMAP Tx Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:59:29 +00:00
static void
ena_release_all_tx_dmamap(struct ena_ring *tx_ring)
{
struct ena_adapter *adapter = tx_ring->adapter;
struct ena_tx_buffer *tx_info;
bus_dma_tag_t tx_tag = adapter->tx_buf_tag;;
int i;
#ifdef DEV_NETMAP
struct ena_netmap_tx_info *nm_info;
int j;
#endif /* DEV_NETMAP */
for (i = 0; i < tx_ring->ring_size; ++i) {
tx_info = &tx_ring->tx_buffer_info[i];
#ifdef DEV_NETMAP
if (adapter->ifp->if_capenable & IFCAP_NETMAP) {
nm_info = &tx_info->nm_info;
for (j = 0; j < ENA_PKT_MAX_BUFS; ++j) {
if (nm_info->map_seg[j] != NULL) {
bus_dmamap_destroy(tx_tag,
nm_info->map_seg[j]);
nm_info->map_seg[j] = NULL;
}
}
}
#endif /* DEV_NETMAP */
Rework and simplify Tx DMA mapping in ENA Driver working in LLQ mode in some cases can send only few last segments of the mbuf using DMA engine, and the rest of them are sent to the device using direct PCI transaction. To map the only necessary data, two DMA maps were used. That solution was very rough and was causing a bug - if both maps were used (head_map and seg_map), there was a race in between two flows on two queues and the device was receiving corrupted data which could be further received on the other host if the Tx cksum offload was enabled. As it's ok to map whole mbuf and then send to the device only needed segments, the design was simplified to use only single DMA map. The driver version was updated to v2.1.1 as it's important bug fix. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc.
2020-02-24 15:35:31 +00:00
if (tx_info->dmamap != NULL) {
bus_dmamap_destroy(tx_tag, tx_info->dmamap);
tx_info->dmamap = NULL;
Add support for ENA NETMAP Tx Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:59:29 +00:00
}
}
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/**
* ena_setup_tx_resources - allocate Tx resources (Descriptors)
* @adapter: network interface device structure
* @qid: queue index
*
* Returns 0 on success, otherwise on failure.
**/
static int
ena_setup_tx_resources(struct ena_adapter *adapter, int qid)
{
struct ena_que *que = &adapter->que[qid];
struct ena_ring *tx_ring = que->tx_ring;
int size, i, err;
Add support for ENA NETMAP Tx Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:59:29 +00:00
#ifdef DEV_NETMAP
bus_dmamap_t *map;
int j;
ena_netmap_reset_tx_ring(adapter, qid);
#endif /* DEV_NETMAP */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
size = sizeof(struct ena_tx_buffer) * tx_ring->ring_size;
tx_ring->tx_buffer_info = malloc(size, M_DEVBUF, M_NOWAIT | M_ZERO);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(tx_ring->tx_buffer_info == NULL))
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
return (ENOMEM);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
size = sizeof(uint16_t) * tx_ring->ring_size;
tx_ring->free_tx_ids = malloc(size, M_DEVBUF, M_NOWAIT | M_ZERO);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(tx_ring->free_tx_ids == NULL))
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
goto err_buf_info_free;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
size = tx_ring->tx_max_header_size;
tx_ring->push_buf_intermediate_buf = malloc(size, M_DEVBUF,
M_NOWAIT | M_ZERO);
if (unlikely(tx_ring->push_buf_intermediate_buf == NULL))
goto err_tx_ids_free;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Req id stack for TX OOO completions */
for (i = 0; i < tx_ring->ring_size; i++)
tx_ring->free_tx_ids[i] = i;
/* Reset TX statistics. */
ena_reset_counters((counter_u64_t *)&tx_ring->tx_stats,
sizeof(tx_ring->tx_stats));
tx_ring->next_to_use = 0;
tx_ring->next_to_clean = 0;
Add additional doorbells on ENA Tx path The new ENA HAL is introducing API, which can determine on Tx path if the doorbell is needed. That way, it can tell the driver, that it should call an doorbell. The old threshold value wasn't removed, as not all HW is supporting this feature - so it was reworked to also work with the new API. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:33:31 +00:00
tx_ring->acum_pkts = 0;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Make sure that drbr is empty */
Acquire locks before calling drbr_flush() It is required to hold lock that is associated with buffer ring before flushing drbr. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-04 00:00:42 +00:00
ENA_RING_MTX_LOCK(tx_ring);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
drbr_flush(adapter->ifp, tx_ring->br);
Acquire locks before calling drbr_flush() It is required to hold lock that is associated with buffer ring before flushing drbr. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-04 00:00:42 +00:00
ENA_RING_MTX_UNLOCK(tx_ring);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* ... and create the buffer DMA maps */
for (i = 0; i < tx_ring->ring_size; i++) {
err = bus_dmamap_create(adapter->tx_buf_tag, 0,
Rework and simplify Tx DMA mapping in ENA Driver working in LLQ mode in some cases can send only few last segments of the mbuf using DMA engine, and the rest of them are sent to the device using direct PCI transaction. To map the only necessary data, two DMA maps were used. That solution was very rough and was causing a bug - if both maps were used (head_map and seg_map), there was a race in between two flows on two queues and the device was receiving corrupted data which could be further received on the other host if the Tx cksum offload was enabled. As it's ok to map whole mbuf and then send to the device only needed segments, the design was simplified to use only single DMA map. The driver version was updated to v2.1.1 as it's important bug fix. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc.
2020-02-24 15:35:31 +00:00
&tx_ring->tx_buffer_info[i].dmamap);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(err != 0)) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_ALERT,
Rework and simplify Tx DMA mapping in ENA Driver working in LLQ mode in some cases can send only few last segments of the mbuf using DMA engine, and the rest of them are sent to the device using direct PCI transaction. To map the only necessary data, two DMA maps were used. That solution was very rough and was causing a bug - if both maps were used (head_map and seg_map), there was a race in between two flows on two queues and the device was receiving corrupted data which could be further received on the other host if the Tx cksum offload was enabled. As it's ok to map whole mbuf and then send to the device only needed segments, the design was simplified to use only single DMA map. The driver version was updated to v2.1.1 as it's important bug fix. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc.
2020-02-24 15:35:31 +00:00
"Unable to create Tx DMA map for buffer %d\n",
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
i);
Add support for ENA NETMAP Tx Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:59:29 +00:00
goto err_map_release;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Add support for ENA NETMAP Tx Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:59:29 +00:00
#ifdef DEV_NETMAP
if (adapter->ifp->if_capenable & IFCAP_NETMAP) {
map = tx_ring->tx_buffer_info[i].nm_info.map_seg;
for (j = 0; j < ENA_PKT_MAX_BUFS; j++) {
err = bus_dmamap_create(adapter->tx_buf_tag, 0,
&map[j]);
if (unlikely(err != 0)) {
ena_trace(ENA_ALERT, "Unable to create "
"Tx DMA for buffer %d %d\n", i, j);
goto err_map_release;
}
}
}
#endif /* DEV_NETMAP */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
/* Allocate taskqueues */
TASK_INIT(&tx_ring->enqueue_task, 0, ena_deferred_mq_start, tx_ring);
tx_ring->enqueue_tq = taskqueue_create_fast("ena_tx_enque", M_NOWAIT,
taskqueue_thread_enqueue, &tx_ring->enqueue_tq);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(tx_ring->enqueue_tq == NULL)) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_ALERT,
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
"Unable to create taskqueue for enqueue task\n");
i = tx_ring->ring_size;
Add support for ENA NETMAP Tx Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:59:29 +00:00
goto err_map_release;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Lock optimization in ENA Handle IO interrupts using filter routine. That way, the main cleanup task could be moved to the separate thread using taskqueue. The deferred Rx cleanup task was removed, and now the cleanup task is begin called instead. That way, the Rx lock could be removed. In addition, Queue management (wake up and stop TX ring) was added, so the TX cleanup task can be performed mostly lockless. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:29:24 +00:00
tx_ring->running = true;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
taskqueue_start_threads(&tx_ring->enqueue_tq, 1, PI_NET,
"%s txeq %d", device_get_nameunit(adapter->pdev), que->cpu);
return (0);
Add support for ENA NETMAP Tx Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:59:29 +00:00
err_map_release:
ena_release_all_tx_dmamap(tx_ring);
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
err_tx_ids_free:
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
free(tx_ring->free_tx_ids, M_DEVBUF);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
tx_ring->free_tx_ids = NULL;
err_buf_info_free:
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
free(tx_ring->tx_buffer_info, M_DEVBUF);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
tx_ring->tx_buffer_info = NULL;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (ENOMEM);
}
/**
* ena_free_tx_resources - Free Tx Resources per Queue
* @adapter: network interface device structure
* @qid: queue index
*
* Free all transmit software resources
**/
static void
ena_free_tx_resources(struct ena_adapter *adapter, int qid)
{
struct ena_ring *tx_ring = &adapter->tx_ring[qid];
Add support for ENA NETMAP Tx Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:59:29 +00:00
#ifdef DEV_NETMAP
struct ena_netmap_tx_info *nm_info;
int j;
#endif /* DEV_NETMAP */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
while (taskqueue_cancel(tx_ring->enqueue_tq, &tx_ring->enqueue_task,
NULL))
taskqueue_drain(tx_ring->enqueue_tq, &tx_ring->enqueue_task);
taskqueue_free(tx_ring->enqueue_tq);
Acquire locks before calling drbr_flush() It is required to hold lock that is associated with buffer ring before flushing drbr. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-04 00:00:42 +00:00
ENA_RING_MTX_LOCK(tx_ring);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Flush buffer ring, */
drbr_flush(adapter->ifp, tx_ring->br);
/* Free buffer DMA maps, */
for (int i = 0; i < tx_ring->ring_size; i++) {
Rework and simplify Tx DMA mapping in ENA Driver working in LLQ mode in some cases can send only few last segments of the mbuf using DMA engine, and the rest of them are sent to the device using direct PCI transaction. To map the only necessary data, two DMA maps were used. That solution was very rough and was causing a bug - if both maps were used (head_map and seg_map), there was a race in between two flows on two queues and the device was receiving corrupted data which could be further received on the other host if the Tx cksum offload was enabled. As it's ok to map whole mbuf and then send to the device only needed segments, the design was simplified to use only single DMA map. The driver version was updated to v2.1.1 as it's important bug fix. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc.
2020-02-24 15:35:31 +00:00
bus_dmamap_sync(adapter->tx_buf_tag,
tx_ring->tx_buffer_info[i].dmamap, BUS_DMASYNC_POSTWRITE);
bus_dmamap_unload(adapter->tx_buf_tag,
tx_ring->tx_buffer_info[i].dmamap);
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
bus_dmamap_destroy(adapter->tx_buf_tag,
Rework and simplify Tx DMA mapping in ENA Driver working in LLQ mode in some cases can send only few last segments of the mbuf using DMA engine, and the rest of them are sent to the device using direct PCI transaction. To map the only necessary data, two DMA maps were used. That solution was very rough and was causing a bug - if both maps were used (head_map and seg_map), there was a race in between two flows on two queues and the device was receiving corrupted data which could be further received on the other host if the Tx cksum offload was enabled. As it's ok to map whole mbuf and then send to the device only needed segments, the design was simplified to use only single DMA map. The driver version was updated to v2.1.1 as it's important bug fix. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc.
2020-02-24 15:35:31 +00:00
tx_ring->tx_buffer_info[i].dmamap);
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
Add support for ENA NETMAP Tx Two new tables are added to ena_tx_buffer structure: * netmap_map_seg stores DMA mapping structures, * netmap_buf_idx stores buff indexes taken from the slots. When Tx resources are being set, the new mapping structures are created and netmap Tx rings are being reset. When Tx resources are being released, used netmap bufs are unmapped from DMA and then mapping structures are destroyed. When Tx interrupt occurrs, ena_netmap_tx_irq is called. ena_netmap_txsync callback signalizes that there are new packets which should be transmitted. First, it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_tx_frames moves packets from netmap ring to NIC, * ena_netmap_tx_cleanup restores buffers from NIC and gives them back to the userspace app. 0 is returned in case of Tx error that could be handled by the driver. ena_netmap_tx_frames checks if there are packets ready for transmission. Then, for each of them, ena_netmap_tx_frame is called. If error occurs, transmitting is stopped, but if the error was cause due to HW ring being full, information about that is not propagated to the userspace app. When all packets are ready, doorbell is written to NIC and netmap ring state is updated. Parsing of one packet is done by the ena_netmap_tx_frame function. First, it checks if number of slots does not exceed NIC limit. Invalid packets are being dropped and the error is propagated to the upper layer. As each netmap buffer has equal size, which is typically greater then 2KiB, there shouldn't be any packets which contain too many slots. Then, the ena_com_tx_ctx structure is being filled. As netmap does not support any hardware offloads, ena_com_tx_meta structure is set to zero. After that, ena_netmap_map_slots maps all memory slots for DMA. If the device works in the LLQ mode, the push header is being determined by checking if the header fits within the first socket. If so, the portion of data is being copied directly from the slot. In other case, the data is copied to the intermediate buffer. First slots are treated the same as as the others, because DMA mapping has no impact on LLQ mode. Index of each netmap buffer is taken from slot and stored in netmap_buf_idx array. In case of mapping error, memory is unmapped and packets are put back to the netmap ring. ena_netmap_tx_cleanup performs out of order cleanup of sent buffers. First, req_id is taken and is validated. As validate_tx_req_id from ena.c is specific to kernels mbuf, another implementation is provided. Each req_id is cleaned up by ena_netmap_tx_clean_one function. Buffers are being unmaped from DMA and put back to netmap ring. In the end, state of netmap and NIC rings are being updated. Differential Revision: https://reviews.freebsd.org/D21936 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:59:29 +00:00
#ifdef DEV_NETMAP
if (adapter->ifp->if_capenable & IFCAP_NETMAP) {
nm_info = &tx_ring->tx_buffer_info[i].nm_info;
for (j = 0; j < ENA_PKT_MAX_BUFS; j++) {
if (nm_info->socket_buf_idx[j] != 0) {
bus_dmamap_sync(adapter->tx_buf_tag,
nm_info->map_seg[j],
BUS_DMASYNC_POSTWRITE);
ena_netmap_unload(adapter,
nm_info->map_seg[j]);
}
bus_dmamap_destroy(adapter->tx_buf_tag,
nm_info->map_seg[j]);
nm_info->socket_buf_idx[j] = 0;
}
}
#endif /* DEV_NETMAP */
Fix DMA synchronization in the ENA driver Tx and Rx paths The DMA in FreeBSD requires explicit synchronization. ENA driver was only doing PREREAD and PREWRITE synchronizations. Missing bus_dmamap_sync() calls were added. It is also required to synchronize DMA engine before unloading DMA map. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:18:23 +00:00
m_freem(tx_ring->tx_buffer_info[i].mbuf);
tx_ring->tx_buffer_info[i].mbuf = NULL;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Introduce additional locks when releasing TX resources and buffers in ENA There could be race condition with TX cleaning routine when cleaning mbufs, when it was called directly from main sending thread (ena_mq_start). Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10927
2017-05-30 12:00:56 +00:00
ENA_RING_MTX_UNLOCK(tx_ring);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* And free allocated memory. */
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
free(tx_ring->tx_buffer_info, M_DEVBUF);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
tx_ring->tx_buffer_info = NULL;
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
free(tx_ring->free_tx_ids, M_DEVBUF);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
tx_ring->free_tx_ids = NULL;
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
ENA_MEM_FREE(adapter->ena_dev->dmadev,
tx_ring->push_buf_intermediate_buf);
tx_ring->push_buf_intermediate_buf = NULL;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
/**
* ena_setup_all_tx_resources - allocate all queues Tx resources
* @adapter: network interface device structure
*
* Returns 0 on success, otherwise on failure.
**/
static int
ena_setup_all_tx_resources(struct ena_adapter *adapter)
{
int i, rc;
for (i = 0; i < adapter->num_queues; i++) {
rc = ena_setup_tx_resources(adapter, i);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (rc != 0) {
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
device_printf(adapter->pdev,
"Allocation for Tx Queue %u failed\n", i);
goto err_setup_tx;
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
return (0);
err_setup_tx:
/* Rewind the index freeing the rings as we go */
while (i--)
ena_free_tx_resources(adapter, i);
return (rc);
}
/**
* ena_free_all_tx_resources - Free Tx Resources for All Queues
* @adapter: network interface device structure
*
* Free all transmit software resources
**/
static void
ena_free_all_tx_resources(struct ena_adapter *adapter)
{
int i;
for (i = 0; i < adapter->num_queues; i++)
ena_free_tx_resources(adapter, i);
}
/**
* ena_setup_rx_resources - allocate Rx resources (Descriptors)
* @adapter: network interface device structure
* @qid: queue index
*
* Returns 0 on success, otherwise on failure.
**/
static int
ena_setup_rx_resources(struct ena_adapter *adapter, unsigned int qid)
{
struct ena_que *que = &adapter->que[qid];
struct ena_ring *rx_ring = que->rx_ring;
int size, err, i;
size = sizeof(struct ena_rx_buffer) * rx_ring->ring_size;
Add support for ENA NETMAP Rx Most of code used for Rx ring initialization could be reused in NETMAP. Reset of NETMAP ring and new alloc method was added. Driver decides if use kernels mbufs or NETMAPs slots based on IFCAP_NETMAP flag. It allows to reuse ena_refill_rx_bufs, which provides proper handling of Rx out of order completion. ena_netmap_alloc_rx_slot takes exactly the same arguments as ena_alloc_rx_mbuf, but instead of allocating one mbuf it takes one slot from NETMAP ring. Based on queue id proper netmap_ring is found. As NETMAP provides the "partial opening" feature not all of the rings are avaiable. Not used points to invalid ring. If there is available slot, it is taken from the ring. Its buffer is mapped to DMA and its index is stored in ena_rx_buffer field in ena_rx_buffer structure. Then ena_buf is filled with addresses and ring state is updated. Cleanup is handled by ena_netmap_free_rx_slot. It unmaps DMA and returns buffer to ring. As we could not return more bufs than we have taken and we should not override occupied slots, buf_index should be 0. It is being checked by assertion. ena_netmap_rxsync callback puts received packets back to NETMAP ring and passes them to user space by updating ring pointers. First it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_rx_frames moves received frames from NIC to NETMAP ring, * ena_netmap_rx_cleanup fills NIC ring with slots released by userspace app. In case of Rx error that could be handled by NIC driver (for example by performing reset) rx sync should return 0. ena_netmap_rx_frames first checks if NETMAP ring is in consistent state and then in the loop receives new frames. When all available frames are taken nr_hwtail is updated. Receiving one frame is handled by ena_netmap_rx_frame. If no error occurrs, each Descriptor is loaded by ena_netmap_rx_load_desc function. If packets take more than one segments NS_MOREFRAG flag must be set in all, but not last slot. In case of wrong req_id packet is removed from NETMAP ring. If packet is successful received counters are updated. Refiling of NIC ring is performed by ena_netmap_rx_cleanup function. It calculates number of available slots and call ena_refill_rx_bufs with proper number. Differential Revision: https://reviews.freebsd.org/D21935 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:57:44 +00:00
#ifdef DEV_NETMAP
ena_netmap_reset_rx_ring(adapter, qid);
rx_ring->initialized = false;
#endif /* DEV_NETMAP */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/*
* Alloc extra element so in rx path
* we can always prefetch rx_info + 1
*/
size += sizeof(struct ena_rx_buffer);
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
rx_ring->rx_buffer_info = malloc(size, M_DEVBUF, M_WAITOK | M_ZERO);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Add RX OOO completion feature The RX out of order completion feature, allows to complete RX descriptors out of order, by keeping trace of all free descriptors in the separate array. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12855
2017-11-09 11:45:59 +00:00
size = sizeof(uint16_t) * rx_ring->ring_size;
rx_ring->free_rx_ids = malloc(size, M_DEVBUF, M_WAITOK);
for (i = 0; i < rx_ring->ring_size; i++)
rx_ring->free_rx_ids[i] = i;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Reset RX statistics. */
ena_reset_counters((counter_u64_t *)&rx_ring->rx_stats,
sizeof(rx_ring->rx_stats));
rx_ring->next_to_clean = 0;
rx_ring->next_to_use = 0;
/* ... and create the buffer DMA maps */
for (i = 0; i < rx_ring->ring_size; i++) {
err = bus_dmamap_create(adapter->rx_buf_tag, 0,
&(rx_ring->rx_buffer_info[i].map));
if (err != 0) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_ALERT,
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
"Unable to create Rx DMA map for buffer %d\n", i);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
goto err_buf_info_unmap;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
/* Create LRO for the ring */
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((adapter->ifp->if_capenable & IFCAP_LRO) != 0) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
int err = tcp_lro_init(&rx_ring->lro);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (err != 0) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
"LRO[%d] Initialization failed!\n", qid);
} else {
ena_trace(ENA_INFO,
"RX Soft LRO[%d] Initialized\n", qid);
rx_ring->lro.ifp = adapter->ifp;
}
}
return (0);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
err_buf_info_unmap:
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
while (i--) {
bus_dmamap_destroy(adapter->rx_buf_tag,
rx_ring->rx_buffer_info[i].map);
}
Add RX OOO completion feature The RX out of order completion feature, allows to complete RX descriptors out of order, by keeping trace of all free descriptors in the separate array. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12855
2017-11-09 11:45:59 +00:00
free(rx_ring->free_rx_ids, M_DEVBUF);
rx_ring->free_rx_ids = NULL;
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
free(rx_ring->rx_buffer_info, M_DEVBUF);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rx_ring->rx_buffer_info = NULL;
return (ENOMEM);
}
/**
* ena_free_rx_resources - Free Rx Resources
* @adapter: network interface device structure
* @qid: queue index
*
* Free all receive software resources
**/
static void
ena_free_rx_resources(struct ena_adapter *adapter, unsigned int qid)
{
struct ena_ring *rx_ring = &adapter->rx_ring[qid];
/* Free buffer DMA maps, */
for (int i = 0; i < rx_ring->ring_size; i++) {
Fix DMA synchronization in the ENA driver Tx and Rx paths The DMA in FreeBSD requires explicit synchronization. ENA driver was only doing PREREAD and PREWRITE synchronizations. Missing bus_dmamap_sync() calls were added. It is also required to synchronize DMA engine before unloading DMA map. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:18:23 +00:00
bus_dmamap_sync(adapter->rx_buf_tag,
rx_ring->rx_buffer_info[i].map, BUS_DMASYNC_POSTREAD);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
m_freem(rx_ring->rx_buffer_info[i].mbuf);
rx_ring->rx_buffer_info[i].mbuf = NULL;
bus_dmamap_unload(adapter->rx_buf_tag,
rx_ring->rx_buffer_info[i].map);
bus_dmamap_destroy(adapter->rx_buf_tag,
rx_ring->rx_buffer_info[i].map);
}
/* free LRO resources, */
tcp_lro_free(&rx_ring->lro);
/* free allocated memory */
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
free(rx_ring->rx_buffer_info, M_DEVBUF);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rx_ring->rx_buffer_info = NULL;
Add RX OOO completion feature The RX out of order completion feature, allows to complete RX descriptors out of order, by keeping trace of all free descriptors in the separate array. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12855
2017-11-09 11:45:59 +00:00
free(rx_ring->free_rx_ids, M_DEVBUF);
rx_ring->free_rx_ids = NULL;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
/**
* ena_setup_all_rx_resources - allocate all queues Rx resources
* @adapter: network interface device structure
*
* Returns 0 on success, otherwise on failure.
**/
static int
ena_setup_all_rx_resources(struct ena_adapter *adapter)
{
int i, rc = 0;
for (i = 0; i < adapter->num_queues; i++) {
rc = ena_setup_rx_resources(adapter, i);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (rc != 0) {
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
device_printf(adapter->pdev,
"Allocation for Rx Queue %u failed\n", i);
goto err_setup_rx;
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
return (0);
err_setup_rx:
/* rewind the index freeing the rings as we go */
while (i--)
ena_free_rx_resources(adapter, i);
return (rc);
}
/**
* ena_free_all_rx_resources - Free Rx resources for all queues
* @adapter: network interface device structure
*
* Free all receive software resources
**/
static void
ena_free_all_rx_resources(struct ena_adapter *adapter)
{
int i;
for (i = 0; i < adapter->num_queues; i++)
ena_free_rx_resources(adapter, i);
}
static inline int
ena_alloc_rx_mbuf(struct ena_adapter *adapter,
struct ena_ring *rx_ring, struct ena_rx_buffer *rx_info)
{
struct ena_com_buf *ena_buf;
bus_dma_segment_t segs[1];
int nsegs, error;
Allow usage of more RX descriptors than 1 in ENA driver Using only 1 descriptor on RX could be an issue, if system would be low on resources and could not provide driver with large chunks of contiguous memory. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12871
2017-11-09 13:36:42 +00:00
int mlen;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* if previous allocated frag is not used */
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rx_info->mbuf != NULL))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (0);
/* Get mbuf using UMA allocator */
rx_info->mbuf = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM16BYTES);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rx_info->mbuf == NULL)) {
Allow usage of more RX descriptors than 1 in ENA driver Using only 1 descriptor on RX could be an issue, if system would be low on resources and could not provide driver with large chunks of contiguous memory. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12871
2017-11-09 13:36:42 +00:00
counter_u64_add(rx_ring->rx_stats.mjum_alloc_fail, 1);
rx_info->mbuf = m_getcl(M_NOWAIT, MT_DATA, M_PKTHDR);
if (unlikely(rx_info->mbuf == NULL)) {
counter_u64_add(rx_ring->rx_stats.mbuf_alloc_fail, 1);
return (ENOMEM);
}
mlen = MCLBYTES;
} else {
mlen = MJUM16BYTES;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
/* Set mbuf length*/
Allow usage of more RX descriptors than 1 in ENA driver Using only 1 descriptor on RX could be an issue, if system would be low on resources and could not provide driver with large chunks of contiguous memory. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12871
2017-11-09 13:36:42 +00:00
rx_info->mbuf->m_pkthdr.len = rx_info->mbuf->m_len = mlen;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Map packets for DMA */
ena_trace(ENA_DBG | ENA_RSC | ENA_RXPTH,
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
"Using tag %p for buffers' DMA mapping, mbuf %p len: %d\n",
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
adapter->rx_buf_tag,rx_info->mbuf, rx_info->mbuf->m_len);
error = bus_dmamap_load_mbuf_sg(adapter->rx_buf_tag, rx_info->map,
rx_info->mbuf, segs, &nsegs, BUS_DMA_NOWAIT);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely((error != 0) || (nsegs != 1))) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_WARNING, "failed to map mbuf, error: %d, "
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
"nsegs: %d\n", error, nsegs);
counter_u64_add(rx_ring->rx_stats.dma_mapping_err, 1);
goto exit;
}
bus_dmamap_sync(adapter->rx_buf_tag, rx_info->map, BUS_DMASYNC_PREREAD);
ena_buf = &rx_info->ena_buf;
ena_buf->paddr = segs[0].ds_addr;
Allow usage of more RX descriptors than 1 in ENA driver Using only 1 descriptor on RX could be an issue, if system would be low on resources and could not provide driver with large chunks of contiguous memory. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12871
2017-11-09 13:36:42 +00:00
ena_buf->len = mlen;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_trace(ENA_DBG | ENA_RSC | ENA_RXPTH,
"ALLOC RX BUF: mbuf %p, rx_info %p, len %d, paddr %#jx\n",
rx_info->mbuf, rx_info,ena_buf->len, (uintmax_t)ena_buf->paddr);
return (0);
exit:
m_freem(rx_info->mbuf);
rx_info->mbuf = NULL;
return (EFAULT);
}
static void
ena_free_rx_mbuf(struct ena_adapter *adapter, struct ena_ring *rx_ring,
struct ena_rx_buffer *rx_info)
{
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
if (rx_info->mbuf == NULL) {
ena_trace(ENA_WARNING, "Trying to free unallocated buffer\n");
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return;
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Fix DMA synchronization in the ENA driver Tx and Rx paths The DMA in FreeBSD requires explicit synchronization. ENA driver was only doing PREREAD and PREWRITE synchronizations. Missing bus_dmamap_sync() calls were added. It is also required to synchronize DMA engine before unloading DMA map. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:18:23 +00:00
bus_dmamap_sync(adapter->rx_buf_tag, rx_info->map,
BUS_DMASYNC_POSTREAD);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
bus_dmamap_unload(adapter->rx_buf_tag, rx_info->map);
m_freem(rx_info->mbuf);
rx_info->mbuf = NULL;
}
/**
* ena_refill_rx_bufs - Refills ring with descriptors
* @rx_ring: the ring which we want to feed with free descriptors
* @num: number of descriptors to refill
* Refills the ring with newly allocated DMA-mapped mbufs for receiving
**/
Split Rx/Tx from initialization code in ENA driver Move Rx/Tx routines to separate file. Some functions: * ena_restore_device, * ena_destroy_device, * ena_up, * ena_down, * ena_refill_rx_bufs could be reused in upcoming netmap code in the driver. To make it possible, they were moved to ena.h header. Differential Revision: https://reviews.freebsd.org/D21933 Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:44:26 +00:00
int
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_refill_rx_bufs(struct ena_ring *rx_ring, uint32_t num)
{
struct ena_adapter *adapter = rx_ring->adapter;
Add RX OOO completion feature The RX out of order completion feature, allows to complete RX descriptors out of order, by keeping trace of all free descriptors in the separate array. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12855
2017-11-09 11:45:59 +00:00
uint16_t next_to_use, req_id;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
uint32_t i;
int rc;
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
ena_trace(ENA_DBG | ENA_RXPTH | ENA_RSC, "refill qid: %d\n",
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rx_ring->qid);
next_to_use = rx_ring->next_to_use;
for (i = 0; i < num; i++) {
Add RX OOO completion feature The RX out of order completion feature, allows to complete RX descriptors out of order, by keeping trace of all free descriptors in the separate array. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12855
2017-11-09 11:45:59 +00:00
struct ena_rx_buffer *rx_info;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_trace(ENA_DBG | ENA_RXPTH | ENA_RSC,
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
"RX buffer - next to use: %d\n", next_to_use);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Add RX OOO completion feature The RX out of order completion feature, allows to complete RX descriptors out of order, by keeping trace of all free descriptors in the separate array. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12855
2017-11-09 11:45:59 +00:00
req_id = rx_ring->free_rx_ids[next_to_use];
rx_info = &rx_ring->rx_buffer_info[req_id];
Add support for ENA NETMAP Rx Most of code used for Rx ring initialization could be reused in NETMAP. Reset of NETMAP ring and new alloc method was added. Driver decides if use kernels mbufs or NETMAPs slots based on IFCAP_NETMAP flag. It allows to reuse ena_refill_rx_bufs, which provides proper handling of Rx out of order completion. ena_netmap_alloc_rx_slot takes exactly the same arguments as ena_alloc_rx_mbuf, but instead of allocating one mbuf it takes one slot from NETMAP ring. Based on queue id proper netmap_ring is found. As NETMAP provides the "partial opening" feature not all of the rings are avaiable. Not used points to invalid ring. If there is available slot, it is taken from the ring. Its buffer is mapped to DMA and its index is stored in ena_rx_buffer field in ena_rx_buffer structure. Then ena_buf is filled with addresses and ring state is updated. Cleanup is handled by ena_netmap_free_rx_slot. It unmaps DMA and returns buffer to ring. As we could not return more bufs than we have taken and we should not override occupied slots, buf_index should be 0. It is being checked by assertion. ena_netmap_rxsync callback puts received packets back to NETMAP ring and passes them to user space by updating ring pointers. First it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_rx_frames moves received frames from NIC to NETMAP ring, * ena_netmap_rx_cleanup fills NIC ring with slots released by userspace app. In case of Rx error that could be handled by NIC driver (for example by performing reset) rx sync should return 0. ena_netmap_rx_frames first checks if NETMAP ring is in consistent state and then in the loop receives new frames. When all available frames are taken nr_hwtail is updated. Receiving one frame is handled by ena_netmap_rx_frame. If no error occurrs, each Descriptor is loaded by ena_netmap_rx_load_desc function. If packets take more than one segments NS_MOREFRAG flag must be set in all, but not last slot. In case of wrong req_id packet is removed from NETMAP ring. If packet is successful received counters are updated. Refiling of NIC ring is performed by ena_netmap_rx_cleanup function. It calculates number of available slots and call ena_refill_rx_bufs with proper number. Differential Revision: https://reviews.freebsd.org/D21935 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:57:44 +00:00
#ifdef DEV_NETMAP
Add support for ENA NETMAP partial initialization In NETMAP mode not all queues need to be allocated to NETMAP. Some of them could be left to the kernel. Configuration is managed by the flags nr_mode and nr_pending_mode provided per each NETMAP kring. ENA driver checks those flags and perform proper rings initialization. Differential Revision: https://reviews.freebsd.org/D21937 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 16:02:42 +00:00
if (ena_rx_ring_in_netmap(adapter, rx_ring->qid))
Add support for ENA NETMAP Rx Most of code used for Rx ring initialization could be reused in NETMAP. Reset of NETMAP ring and new alloc method was added. Driver decides if use kernels mbufs or NETMAPs slots based on IFCAP_NETMAP flag. It allows to reuse ena_refill_rx_bufs, which provides proper handling of Rx out of order completion. ena_netmap_alloc_rx_slot takes exactly the same arguments as ena_alloc_rx_mbuf, but instead of allocating one mbuf it takes one slot from NETMAP ring. Based on queue id proper netmap_ring is found. As NETMAP provides the "partial opening" feature not all of the rings are avaiable. Not used points to invalid ring. If there is available slot, it is taken from the ring. Its buffer is mapped to DMA and its index is stored in ena_rx_buffer field in ena_rx_buffer structure. Then ena_buf is filled with addresses and ring state is updated. Cleanup is handled by ena_netmap_free_rx_slot. It unmaps DMA and returns buffer to ring. As we could not return more bufs than we have taken and we should not override occupied slots, buf_index should be 0. It is being checked by assertion. ena_netmap_rxsync callback puts received packets back to NETMAP ring and passes them to user space by updating ring pointers. First it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_rx_frames moves received frames from NIC to NETMAP ring, * ena_netmap_rx_cleanup fills NIC ring with slots released by userspace app. In case of Rx error that could be handled by NIC driver (for example by performing reset) rx sync should return 0. ena_netmap_rx_frames first checks if NETMAP ring is in consistent state and then in the loop receives new frames. When all available frames are taken nr_hwtail is updated. Receiving one frame is handled by ena_netmap_rx_frame. If no error occurrs, each Descriptor is loaded by ena_netmap_rx_load_desc function. If packets take more than one segments NS_MOREFRAG flag must be set in all, but not last slot. In case of wrong req_id packet is removed from NETMAP ring. If packet is successful received counters are updated. Refiling of NIC ring is performed by ena_netmap_rx_cleanup function. It calculates number of available slots and call ena_refill_rx_bufs with proper number. Differential Revision: https://reviews.freebsd.org/D21935 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:57:44 +00:00
rc = ena_netmap_alloc_rx_slot(adapter, rx_ring, rx_info);
else
#endif /* DEV_NETMAP */
rc = ena_alloc_rx_mbuf(adapter, rx_ring, rx_info);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_WARNING,
"failed to alloc buffer for rx queue %d\n",
rx_ring->qid);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
break;
}
rc = ena_com_add_single_rx_desc(rx_ring->ena_com_io_sq,
Add RX OOO completion feature The RX out of order completion feature, allows to complete RX descriptors out of order, by keeping trace of all free descriptors in the separate array. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12855
2017-11-09 11:45:59 +00:00
&rx_info->ena_buf, req_id);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (unlikely(rc != 0)) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_WARNING,
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
"failed to add buffer for rx queue %d\n",
rx_ring->qid);
break;
}
next_to_use = ENA_RX_RING_IDX_NEXT(next_to_use,
rx_ring->ring_size);
}
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(i < num)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
counter_u64_add(rx_ring->rx_stats.refil_partial, 1);
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_WARNING,
"refilled rx qid %d with only %d mbufs (from %d)\n",
rx_ring->qid, i, num);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (likely(i != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
wmb();
ena_com_write_sq_doorbell(rx_ring->ena_com_io_sq);
}
rx_ring->next_to_use = next_to_use;
return (i);
}
static void
ena_free_rx_bufs(struct ena_adapter *adapter, unsigned int qid)
{
struct ena_ring *rx_ring = &adapter->rx_ring[qid];
unsigned int i;
for (i = 0; i < rx_ring->ring_size; i++) {
struct ena_rx_buffer *rx_info = &rx_ring->rx_buffer_info[i];
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (rx_info->mbuf != NULL)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_free_rx_mbuf(adapter, rx_ring, rx_info);
Add support for ENA NETMAP Rx Most of code used for Rx ring initialization could be reused in NETMAP. Reset of NETMAP ring and new alloc method was added. Driver decides if use kernels mbufs or NETMAPs slots based on IFCAP_NETMAP flag. It allows to reuse ena_refill_rx_bufs, which provides proper handling of Rx out of order completion. ena_netmap_alloc_rx_slot takes exactly the same arguments as ena_alloc_rx_mbuf, but instead of allocating one mbuf it takes one slot from NETMAP ring. Based on queue id proper netmap_ring is found. As NETMAP provides the "partial opening" feature not all of the rings are avaiable. Not used points to invalid ring. If there is available slot, it is taken from the ring. Its buffer is mapped to DMA and its index is stored in ena_rx_buffer field in ena_rx_buffer structure. Then ena_buf is filled with addresses and ring state is updated. Cleanup is handled by ena_netmap_free_rx_slot. It unmaps DMA and returns buffer to ring. As we could not return more bufs than we have taken and we should not override occupied slots, buf_index should be 0. It is being checked by assertion. ena_netmap_rxsync callback puts received packets back to NETMAP ring and passes them to user space by updating ring pointers. First it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_rx_frames moves received frames from NIC to NETMAP ring, * ena_netmap_rx_cleanup fills NIC ring with slots released by userspace app. In case of Rx error that could be handled by NIC driver (for example by performing reset) rx sync should return 0. ena_netmap_rx_frames first checks if NETMAP ring is in consistent state and then in the loop receives new frames. When all available frames are taken nr_hwtail is updated. Receiving one frame is handled by ena_netmap_rx_frame. If no error occurrs, each Descriptor is loaded by ena_netmap_rx_load_desc function. If packets take more than one segments NS_MOREFRAG flag must be set in all, but not last slot. In case of wrong req_id packet is removed from NETMAP ring. If packet is successful received counters are updated. Refiling of NIC ring is performed by ena_netmap_rx_cleanup function. It calculates number of available slots and call ena_refill_rx_bufs with proper number. Differential Revision: https://reviews.freebsd.org/D21935 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:57:44 +00:00
#ifdef DEV_NETMAP
if (((if_getflags(adapter->ifp) & IFF_DYING) == 0) &&
(adapter->ifp->if_capenable & IFCAP_NETMAP)) {
if (rx_info->netmap_buf_idx != 0)
ena_netmap_free_rx_slot(adapter, rx_ring,
rx_info);
}
#endif /* DEV_NETMAP */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
/**
* ena_refill_all_rx_bufs - allocate all queues Rx buffers
* @adapter: network interface device structure
*
*/
static void
ena_refill_all_rx_bufs(struct ena_adapter *adapter)
{
struct ena_ring *rx_ring;
int i, rc, bufs_num;
for (i = 0; i < adapter->num_queues; i++) {
rx_ring = &adapter->rx_ring[i];
bufs_num = rx_ring->ring_size - 1;
rc = ena_refill_rx_bufs(rx_ring, bufs_num);
if (unlikely(rc != bufs_num))
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_WARNING, "refilling Queue %d failed. "
"Allocated %d buffers from: %d\n", i, rc, bufs_num);
Add support for ENA NETMAP Rx Most of code used for Rx ring initialization could be reused in NETMAP. Reset of NETMAP ring and new alloc method was added. Driver decides if use kernels mbufs or NETMAPs slots based on IFCAP_NETMAP flag. It allows to reuse ena_refill_rx_bufs, which provides proper handling of Rx out of order completion. ena_netmap_alloc_rx_slot takes exactly the same arguments as ena_alloc_rx_mbuf, but instead of allocating one mbuf it takes one slot from NETMAP ring. Based on queue id proper netmap_ring is found. As NETMAP provides the "partial opening" feature not all of the rings are avaiable. Not used points to invalid ring. If there is available slot, it is taken from the ring. Its buffer is mapped to DMA and its index is stored in ena_rx_buffer field in ena_rx_buffer structure. Then ena_buf is filled with addresses and ring state is updated. Cleanup is handled by ena_netmap_free_rx_slot. It unmaps DMA and returns buffer to ring. As we could not return more bufs than we have taken and we should not override occupied slots, buf_index should be 0. It is being checked by assertion. ena_netmap_rxsync callback puts received packets back to NETMAP ring and passes them to user space by updating ring pointers. First it fills ena_netmap_ctx. Then it performs two actions: * ena_netmap_rx_frames moves received frames from NIC to NETMAP ring, * ena_netmap_rx_cleanup fills NIC ring with slots released by userspace app. In case of Rx error that could be handled by NIC driver (for example by performing reset) rx sync should return 0. ena_netmap_rx_frames first checks if NETMAP ring is in consistent state and then in the loop receives new frames. When all available frames are taken nr_hwtail is updated. Receiving one frame is handled by ena_netmap_rx_frame. If no error occurrs, each Descriptor is loaded by ena_netmap_rx_load_desc function. If packets take more than one segments NS_MOREFRAG flag must be set in all, but not last slot. In case of wrong req_id packet is removed from NETMAP ring. If packet is successful received counters are updated. Refiling of NIC ring is performed by ena_netmap_rx_cleanup function. It calculates number of available slots and call ena_refill_rx_bufs with proper number. Differential Revision: https://reviews.freebsd.org/D21935 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:57:44 +00:00
#ifdef DEV_NETMAP
rx_ring->initialized = true;
#endif /* DEV_NETMAP */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
static void
ena_free_all_rx_bufs(struct ena_adapter *adapter)
{
int i;
for (i = 0; i < adapter->num_queues; i++)
ena_free_rx_bufs(adapter, i);
}
/**
* ena_free_tx_bufs - Free Tx Buffers per Queue
* @adapter: network interface device structure
* @qid: queue index
**/
static void
ena_free_tx_bufs(struct ena_adapter *adapter, unsigned int qid)
{
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
bool print_once = true;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
struct ena_ring *tx_ring = &adapter->tx_ring[qid];
Introduce additional locks when releasing TX resources and buffers in ENA There could be race condition with TX cleaning routine when cleaning mbufs, when it was called directly from main sending thread (ena_mq_start). Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10927
2017-05-30 12:00:56 +00:00
ENA_RING_MTX_LOCK(tx_ring);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
for (int i = 0; i < tx_ring->ring_size; i++) {
struct ena_tx_buffer *tx_info = &tx_ring->tx_buffer_info[i];
if (tx_info->mbuf == NULL)
continue;
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
if (print_once) {
device_printf(adapter->pdev,
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
"free uncompleted tx mbuf qid %d idx 0x%x\n",
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
qid, i);
print_once = false;
} else {
ena_trace(ENA_DBG,
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
"free uncompleted tx mbuf qid %d idx 0x%x\n",
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
qid, i);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Rework and simplify Tx DMA mapping in ENA Driver working in LLQ mode in some cases can send only few last segments of the mbuf using DMA engine, and the rest of them are sent to the device using direct PCI transaction. To map the only necessary data, two DMA maps were used. That solution was very rough and was causing a bug - if both maps were used (head_map and seg_map), there was a race in between two flows on two queues and the device was receiving corrupted data which could be further received on the other host if the Tx cksum offload was enabled. As it's ok to map whole mbuf and then send to the device only needed segments, the design was simplified to use only single DMA map. The driver version was updated to v2.1.1 as it's important bug fix. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf MFC after: 2 weeks Sponsored by: Amazon, Inc.
2020-02-24 15:35:31 +00:00
bus_dmamap_sync(adapter->tx_buf_tag, tx_info->dmamap,
BUS_DMASYNC_POSTWRITE);
bus_dmamap_unload(adapter->tx_buf_tag, tx_info->dmamap);
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
m_free(tx_info->mbuf);
tx_info->mbuf = NULL;
}
Introduce additional locks when releasing TX resources and buffers in ENA There could be race condition with TX cleaning routine when cleaning mbufs, when it was called directly from main sending thread (ena_mq_start). Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10927
2017-05-30 12:00:56 +00:00
ENA_RING_MTX_UNLOCK(tx_ring);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
static void
ena_free_all_tx_bufs(struct ena_adapter *adapter)
{
for (int i = 0; i < adapter->num_queues; i++)
ena_free_tx_bufs(adapter, i);
}
static void
ena_destroy_all_tx_queues(struct ena_adapter *adapter)
{
uint16_t ena_qid;
int i;
for (i = 0; i < adapter->num_queues; i++) {
ena_qid = ENA_IO_TXQ_IDX(i);
ena_com_destroy_io_queue(adapter->ena_dev, ena_qid);
}
}
static void
ena_destroy_all_rx_queues(struct ena_adapter *adapter)
{
uint16_t ena_qid;
int i;
for (i = 0; i < adapter->num_queues; i++) {
ena_qid = ENA_IO_RXQ_IDX(i);
ena_com_destroy_io_queue(adapter->ena_dev, ena_qid);
}
}
static void
ena_destroy_all_io_queues(struct ena_adapter *adapter)
{
Lock optimization in ENA Handle IO interrupts using filter routine. That way, the main cleanup task could be moved to the separate thread using taskqueue. The deferred Rx cleanup task was removed, and now the cleanup task is begin called instead. That way, the Rx lock could be removed. In addition, Queue management (wake up and stop TX ring) was added, so the TX cleanup task can be performed mostly lockless. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:29:24 +00:00
struct ena_que *queue;
int i;
for (i = 0; i < adapter->num_queues; i++) {
queue = &adapter->que[i];
while (taskqueue_cancel(queue->cleanup_tq,
&queue->cleanup_task, NULL))
taskqueue_drain(queue->cleanup_tq,
&queue->cleanup_task);
taskqueue_free(queue->cleanup_tq);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_destroy_all_tx_queues(adapter);
ena_destroy_all_rx_queues(adapter);
}
static int
ena_create_io_queues(struct ena_adapter *adapter)
{
struct ena_com_dev *ena_dev = adapter->ena_dev;
struct ena_com_create_io_ctx ctx;
struct ena_ring *ring;
Lock optimization in ENA Handle IO interrupts using filter routine. That way, the main cleanup task could be moved to the separate thread using taskqueue. The deferred Rx cleanup task was removed, and now the cleanup task is begin called instead. That way, the Rx lock could be removed. In addition, Queue management (wake up and stop TX ring) was added, so the TX cleanup task can be performed mostly lockless. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:29:24 +00:00
struct ena_que *queue;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
uint16_t ena_qid;
uint32_t msix_vector;
int rc, i;
/* Create TX queues */
for (i = 0; i < adapter->num_queues; i++) {
msix_vector = ENA_IO_IRQ_IDX(i);
ena_qid = ENA_IO_TXQ_IDX(i);
ctx.mem_queue_type = ena_dev->tx_mem_queue_type;
ctx.direction = ENA_COM_IO_QUEUE_DIRECTION_TX;
ctx.queue_size = adapter->tx_ring_size;
ctx.msix_vector = msix_vector;
ctx.qid = ena_qid;
rc = ena_com_create_io_queue(ena_dev, &ctx);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (rc != 0) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
"Failed to create io TX queue #%d rc: %d\n", i, rc);
goto err_tx;
}
ring = &adapter->tx_ring[i];
rc = ena_com_get_io_handlers(ena_dev, ena_qid,
&ring->ena_com_io_sq,
&ring->ena_com_io_cq);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (rc != 0) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
"Failed to get TX queue handlers. TX queue num"
" %d rc: %d\n", i, rc);
ena_com_destroy_io_queue(ena_dev, ena_qid);
goto err_tx;
}
}
/* Create RX queues */
for (i = 0; i < adapter->num_queues; i++) {
msix_vector = ENA_IO_IRQ_IDX(i);
ena_qid = ENA_IO_RXQ_IDX(i);
ctx.mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST;
ctx.direction = ENA_COM_IO_QUEUE_DIRECTION_RX;
ctx.queue_size = adapter->rx_ring_size;
ctx.msix_vector = msix_vector;
ctx.qid = ena_qid;
rc = ena_com_create_io_queue(ena_dev, &ctx);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
"Failed to create io RX queue[%d] rc: %d\n", i, rc);
goto err_rx;
}
ring = &adapter->rx_ring[i];
rc = ena_com_get_io_handlers(ena_dev, ena_qid,
&ring->ena_com_io_sq,
&ring->ena_com_io_cq);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
"Failed to get RX queue handlers. RX queue num"
" %d rc: %d\n", i, rc);
ena_com_destroy_io_queue(ena_dev, ena_qid);
goto err_rx;
}
}
Lock optimization in ENA Handle IO interrupts using filter routine. That way, the main cleanup task could be moved to the separate thread using taskqueue. The deferred Rx cleanup task was removed, and now the cleanup task is begin called instead. That way, the Rx lock could be removed. In addition, Queue management (wake up and stop TX ring) was added, so the TX cleanup task can be performed mostly lockless. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:29:24 +00:00
for (i = 0; i < adapter->num_queues; i++) {
queue = &adapter->que[i];
Use NET_TASK_INIT() and NET_GROUPTASK_INIT() for drivers that process incoming packets in taskqueue context. Reviewed by: hselasky Differential Revision: https://reviews.freebsd.org/D23518
2020-02-11 18:57:07 +00:00
NET_TASK_INIT(&queue->cleanup_task, 0, ena_cleanup, queue);
Lock optimization in ENA Handle IO interrupts using filter routine. That way, the main cleanup task could be moved to the separate thread using taskqueue. The deferred Rx cleanup task was removed, and now the cleanup task is begin called instead. That way, the Rx lock could be removed. In addition, Queue management (wake up and stop TX ring) was added, so the TX cleanup task can be performed mostly lockless. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:29:24 +00:00
queue->cleanup_tq = taskqueue_create_fast("ena cleanup",
M_WAITOK, taskqueue_thread_enqueue, &queue->cleanup_tq);
taskqueue_start_threads(&queue->cleanup_tq, 1, PI_NET,
"%s queue %d cleanup",
device_get_nameunit(adapter->pdev), i);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (0);
err_rx:
while (i--)
ena_com_destroy_io_queue(ena_dev, ENA_IO_RXQ_IDX(i));
i = adapter->num_queues;
err_tx:
while (i--)
ena_com_destroy_io_queue(ena_dev, ENA_IO_TXQ_IDX(i));
return (ENXIO);
}
/*********************************************************************
*
* MSIX & Interrupt Service routine
*
**********************************************************************/
/**
* ena_handle_msix - MSIX Interrupt Handler for admin/async queue
* @arg: interrupt number
**/
static void
ena_intr_msix_mgmnt(void *arg)
{
struct ena_adapter *adapter = (struct ena_adapter *)arg;
ena_com_admin_q_comp_intr_handler(adapter->ena_dev);
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (likely(ENA_FLAG_ISSET(ENA_FLAG_DEVICE_RUNNING, adapter)))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_com_aenq_intr_handler(adapter->ena_dev, arg);
}
Lock optimization in ENA Handle IO interrupts using filter routine. That way, the main cleanup task could be moved to the separate thread using taskqueue. The deferred Rx cleanup task was removed, and now the cleanup task is begin called instead. That way, the Rx lock could be removed. In addition, Queue management (wake up and stop TX ring) was added, so the TX cleanup task can be performed mostly lockless. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:29:24 +00:00
/**
* ena_handle_msix - MSIX Interrupt Handler for Tx/Rx
* @arg: queue
**/
static int
ena_handle_msix(void *arg)
{
struct ena_que *queue = arg;
struct ena_adapter *adapter = queue->adapter;
if_t ifp = adapter->ifp;
if (unlikely((if_getdrvflags(ifp) & IFF_DRV_RUNNING) == 0))
return (FILTER_STRAY);
taskqueue_enqueue(queue->cleanup_tq, &queue->cleanup_task);
return (FILTER_HANDLED);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
static int
ena_enable_msix(struct ena_adapter *adapter)
{
device_t dev = adapter->pdev;
Allow partial MSI-x allocation in ENA driver The situation, where part of the MSI-x was not configured properly, was not properly handled. Now, the driver reduces number of queues to reflect number of existing and properly configured MSI-x vectors. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12863
2017-11-09 12:03:06 +00:00
int msix_vecs, msix_req;
int i, rc = 0;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (ENA_FLAG_ISSET(ENA_FLAG_MSIX_ENABLED, adapter)) {
device_printf(dev, "Error, MSI-X is already enabled\n");
return (EINVAL);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Reserved the max msix vectors we might need */
msix_vecs = ENA_MAX_MSIX_VEC(adapter->num_queues);
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
adapter->msix_entries = malloc(msix_vecs * sizeof(struct msix_entry),
M_DEVBUF, M_WAITOK | M_ZERO);
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
ena_trace(ENA_DBG, "trying to enable MSI-X, vectors: %d\n", msix_vecs);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
for (i = 0; i < msix_vecs; i++) {
adapter->msix_entries[i].entry = i;
/* Vectors must start from 1 */
adapter->msix_entries[i].vector = i + 1;
}
Allow partial MSI-x allocation in ENA driver The situation, where part of the MSI-x was not configured properly, was not properly handled. Now, the driver reduces number of queues to reflect number of existing and properly configured MSI-x vectors. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12863
2017-11-09 12:03:06 +00:00
msix_req = msix_vecs;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rc = pci_alloc_msix(dev, &msix_vecs);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(dev,
"Failed to enable MSIX, vectors %d rc %d\n", msix_vecs, rc);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rc = ENOSPC;
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
goto err_msix_free;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Allow partial MSI-x allocation in ENA driver The situation, where part of the MSI-x was not configured properly, was not properly handled. Now, the driver reduces number of queues to reflect number of existing and properly configured MSI-x vectors. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12863
2017-11-09 12:03:06 +00:00
if (msix_vecs != msix_req) {
Check for number of MSI-x upon partial allocation in ENA The ENA driver needs at least 2 MSI-x - one for admin queue, and one for IO queues pair. If there were not enough resources to allocate more than one MSI-x, the device should not be attached. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:22:12 +00:00
if (msix_vecs == ENA_ADMIN_MSIX_VEC) {
device_printf(dev,
"Not enough number of MSI-x allocated: %d\n",
msix_vecs);
pci_release_msi(dev);
rc = ENOSPC;
goto err_msix_free;
}
Allow partial MSI-x allocation in ENA driver The situation, where part of the MSI-x was not configured properly, was not properly handled. Now, the driver reduces number of queues to reflect number of existing and properly configured MSI-x vectors. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12863
2017-11-09 12:03:06 +00:00
device_printf(dev, "Enable only %d MSI-x (out of %d), reduce "
"the number of queues\n", msix_vecs, msix_req);
adapter->num_queues = msix_vecs - ENA_ADMIN_MSIX_VEC;
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
adapter->msix_vecs = msix_vecs;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ENA_FLAG_SET_ATOMIC(ENA_FLAG_MSIX_ENABLED, adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
return (0);
err_msix_free:
free(adapter->msix_entries, M_DEVBUF);
adapter->msix_entries = NULL;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (rc);
}
static void
ena_setup_mgmnt_intr(struct ena_adapter *adapter)
{
snprintf(adapter->irq_tbl[ENA_MGMNT_IRQ_IDX].name,
ENA_IRQNAME_SIZE, "ena-mgmnt@pci:%s",
device_get_nameunit(adapter->pdev));
/*
* Handler is NULL on purpose, it will be set
* when mgmnt interrupt is acquired
*/
adapter->irq_tbl[ENA_MGMNT_IRQ_IDX].handler = NULL;
adapter->irq_tbl[ENA_MGMNT_IRQ_IDX].data = adapter;
adapter->irq_tbl[ENA_MGMNT_IRQ_IDX].vector =
adapter->msix_entries[ENA_MGMNT_IRQ_IDX].vector;
}
Fix NULL pointer dereference in ena_up() If the call to ena_up() in ena_restore_device() fails, next usage of `ifconfig up` will cause NULL pointer dereference. This patch adds additional checks to prevent that. Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:42:52 +00:00
static int
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_setup_io_intr(struct ena_adapter *adapter)
{
static int last_bind_cpu = -1;
int irq_idx;
Fix NULL pointer dereference in ena_up() If the call to ena_up() in ena_restore_device() fails, next usage of `ifconfig up` will cause NULL pointer dereference. This patch adds additional checks to prevent that. Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:42:52 +00:00
if (adapter->msix_entries == NULL)
return (EINVAL);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
for (int i = 0; i < adapter->num_queues; i++) {
irq_idx = ENA_IO_IRQ_IDX(i);
snprintf(adapter->irq_tbl[irq_idx].name, ENA_IRQNAME_SIZE,
"%s-TxRx-%d", device_get_nameunit(adapter->pdev), i);
adapter->irq_tbl[irq_idx].handler = ena_handle_msix;
adapter->irq_tbl[irq_idx].data = &adapter->que[i];
adapter->irq_tbl[irq_idx].vector =
adapter->msix_entries[irq_idx].vector;
ena_trace(ENA_INFO | ENA_IOQ, "ena_setup_io_intr vector: %d\n",
adapter->msix_entries[irq_idx].vector);
Remove RSS support in ENA Receive Side Scaling is optional feature that could be enabled in kernel configuration by defining flag RSS. Kernel uses hash to store and find protocol control block which is stored in hash tables. Kernel and NIC hash functions must be consistent. Otherwise case lookup fails. To achieve this kernel provides API to set proper hash key to NIC. As it is not possible to change key for virtual ENA NIC, this driver cannot support RSS function. ENA is designed to work in virtual environments so supporting hardware version of this card is unnecessary. Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:12:14 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/*
Remove RSS support in ENA Receive Side Scaling is optional feature that could be enabled in kernel configuration by defining flag RSS. Kernel uses hash to store and find protocol control block which is stored in hash tables. Kernel and NIC hash functions must be consistent. Otherwise case lookup fails. To achieve this kernel provides API to set proper hash key to NIC. As it is not possible to change key for virtual ENA NIC, this driver cannot support RSS function. ENA is designed to work in virtual environments so supporting hardware version of this card is unnecessary. Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:12:14 +00:00
* We want to bind rings to the corresponding cpu
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
* using something similar to the RSS round-robin technique.
*/
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(last_bind_cpu < 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
last_bind_cpu = CPU_FIRST();
adapter->que[i].cpu = adapter->irq_tbl[irq_idx].cpu =
last_bind_cpu;
last_bind_cpu = CPU_NEXT(last_bind_cpu);
}
Fix NULL pointer dereference in ena_up() If the call to ena_up() in ena_restore_device() fails, next usage of `ifconfig up` will cause NULL pointer dereference. This patch adds additional checks to prevent that. Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:42:52 +00:00
return (0);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
static int
ena_request_mgmnt_irq(struct ena_adapter *adapter)
{
struct ena_irq *irq;
unsigned long flags;
int rc, rcc;
flags = RF_ACTIVE | RF_SHAREABLE;
irq = &adapter->irq_tbl[ENA_MGMNT_IRQ_IDX];
irq->res = bus_alloc_resource_any(adapter->pdev, SYS_RES_IRQ,
&irq->vector, flags);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(irq->res == NULL)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "could not allocate "
"irq vector: %d\n", irq->vector);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
return (ENXIO);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
rc = bus_setup_intr(adapter->pdev, irq->res,
INTR_TYPE_NET | INTR_MPSAFE, NULL, ena_intr_msix_mgmnt,
irq->data, &irq->cookie);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "failed to register "
"interrupt handler for irq %ju: %d\n",
rman_get_start(irq->res), rc);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
goto err_res_free;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
irq->requested = true;
return (rc);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
err_res_free:
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_INFO | ENA_ADMQ, "releasing resource for irq %d\n",
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
irq->vector);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rcc = bus_release_resource(adapter->pdev, SYS_RES_IRQ,
irq->vector, irq->res);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rcc != 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "dev has no parent while "
"releasing res for irq: %d\n", irq->vector);
irq->res = NULL;
return (rc);
}
static int
ena_request_io_irq(struct ena_adapter *adapter)
{
struct ena_irq *irq;
unsigned long flags = 0;
int rc = 0, i, rcc;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (unlikely(!ENA_FLAG_ISSET(ENA_FLAG_MSIX_ENABLED, adapter))) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
device_printf(adapter->pdev,
"failed to request I/O IRQ: MSI-X is not enabled\n");
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (EINVAL);
} else {
flags = RF_ACTIVE | RF_SHAREABLE;
}
for (i = ENA_IO_IRQ_FIRST_IDX; i < adapter->msix_vecs; i++) {
irq = &adapter->irq_tbl[i];
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(irq->requested))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
continue;
irq->res = bus_alloc_resource_any(adapter->pdev, SYS_RES_IRQ,
&irq->vector, flags);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(irq->res == NULL)) {
Set error value when allocation of IO irq fails in ENA bus_alloc_resource_any() is not returning error value in case of an error. If the function call fails, the error value was not passed to the ena_up() function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:20:42 +00:00
rc = ENOMEM;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "could not allocate "
"irq vector: %d\n", irq->vector);
goto err;
}
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
rc = bus_setup_intr(adapter->pdev, irq->res,
Lock optimization in ENA Handle IO interrupts using filter routine. That way, the main cleanup task could be moved to the separate thread using taskqueue. The deferred Rx cleanup task was removed, and now the cleanup task is begin called instead. That way, the Rx lock could be removed. In addition, Queue management (wake up and stop TX ring) was added, so the TX cleanup task can be performed mostly lockless. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:29:24 +00:00
INTR_TYPE_NET | INTR_MPSAFE, irq->handler, NULL,
irq->data, &irq->cookie);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "failed to register "
"interrupt handler for irq %ju: %d\n",
rman_get_start(irq->res), rc);
goto err;
}
irq->requested = true;
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_INFO, "queue %d - cpu %d\n",
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
i - ENA_IO_IRQ_FIRST_IDX, irq->cpu);
}
return (rc);
err:
for (; i >= ENA_IO_IRQ_FIRST_IDX; i--) {
irq = &adapter->irq_tbl[i];
rcc = 0;
/* Once we entered err: section and irq->requested is true we
free both intr and resources */
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (irq->requested)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rcc = bus_teardown_intr(adapter->pdev, irq->res, irq->cookie);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rcc != 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "could not release"
" irq: %d, error: %d\n", irq->vector, rcc);
/* If we entred err: section without irq->requested set we know
it was bus_alloc_resource_any() that needs cleanup, provided
res is not NULL. In case res is NULL no work in needed in
this iteration */
rcc = 0;
if (irq->res != NULL) {
rcc = bus_release_resource(adapter->pdev, SYS_RES_IRQ,
irq->vector, irq->res);
}
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rcc != 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "dev has no parent while "
"releasing res for irq: %d\n", irq->vector);
irq->requested = false;
irq->res = NULL;
}
return (rc);
}
static void
ena_free_mgmnt_irq(struct ena_adapter *adapter)
{
struct ena_irq *irq;
int rc;
irq = &adapter->irq_tbl[ENA_MGMNT_IRQ_IDX];
if (irq->requested) {
ena_trace(ENA_INFO | ENA_ADMQ, "tear down irq: %d\n",
irq->vector);
rc = bus_teardown_intr(adapter->pdev, irq->res, irq->cookie);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "failed to tear "
"down irq: %d\n", irq->vector);
irq->requested = 0;
}
if (irq->res != NULL) {
ena_trace(ENA_INFO | ENA_ADMQ, "release resource irq: %d\n",
irq->vector);
rc = bus_release_resource(adapter->pdev, SYS_RES_IRQ,
irq->vector, irq->res);
irq->res = NULL;
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "dev has no parent while "
"releasing res for irq: %d\n", irq->vector);
}
}
static void
ena_free_io_irq(struct ena_adapter *adapter)
{
struct ena_irq *irq;
int rc;
for (int i = ENA_IO_IRQ_FIRST_IDX; i < adapter->msix_vecs; i++) {
irq = &adapter->irq_tbl[i];
if (irq->requested) {
ena_trace(ENA_INFO | ENA_IOQ, "tear down irq: %d\n",
irq->vector);
rc = bus_teardown_intr(adapter->pdev, irq->res,
irq->cookie);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "failed to tear "
"down irq: %d\n", irq->vector);
}
irq->requested = 0;
}
if (irq->res != NULL) {
ena_trace(ENA_INFO | ENA_IOQ, "release resource irq: %d\n",
irq->vector);
rc = bus_release_resource(adapter->pdev, SYS_RES_IRQ,
irq->vector, irq->res);
irq->res = NULL;
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "dev has no parent"
" while releasing res for irq: %d\n",
irq->vector);
}
}
}
}
static void
ena_free_irqs(struct ena_adapter* adapter)
{
ena_free_io_irq(adapter);
ena_free_mgmnt_irq(adapter);
ena_disable_msix(adapter);
}
static void
ena_disable_msix(struct ena_adapter *adapter)
{
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (ENA_FLAG_ISSET(ENA_FLAG_MSIX_ENABLED, adapter)) {
ENA_FLAG_CLEAR_ATOMIC(ENA_FLAG_MSIX_ENABLED, adapter);
pci_release_msi(adapter->pdev);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
adapter->msix_vecs = 0;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (adapter->msix_entries != NULL)
free(adapter->msix_entries, M_DEVBUF);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
adapter->msix_entries = NULL;
}
static void
ena_unmask_all_io_irqs(struct ena_adapter *adapter)
{
struct ena_com_io_cq* io_cq;
struct ena_eth_io_intr_reg intr_reg;
uint16_t ena_qid;
int i;
/* Unmask interrupts for all queues */
for (i = 0; i < adapter->num_queues; i++) {
ena_qid = ENA_IO_TXQ_IDX(i);
io_cq = &adapter->ena_dev->io_cq_queues[ena_qid];
ena_com_update_intr_reg(&intr_reg, 0, 0, true);
ena_com_unmask_intr(io_cq, &intr_reg);
}
}
/* Configure the Rx forwarding */
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
static int
ena_rss_configure(struct ena_adapter *adapter)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
{
struct ena_com_dev *ena_dev = adapter->ena_dev;
int rc;
/* Set indirect table */
rc = ena_com_indirect_table_set(ena_dev);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (unlikely((rc != 0) && (rc != EOPNOTSUPP)))
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Configure hash function (if supported) */
rc = ena_com_set_hash_function(ena_dev);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (unlikely((rc != 0) && (rc != EOPNOTSUPP)))
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Configure hash inputs (if supported) */
rc = ena_com_set_hash_ctrl(ena_dev);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (unlikely((rc != 0) && (rc != EOPNOTSUPP)))
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
return (0);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
static int
ena_up_complete(struct ena_adapter *adapter)
{
int rc;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (likely(ENA_FLAG_ISSET(ENA_FLAG_RSS_ACTIVE, adapter))) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rc = ena_rss_configure(adapter);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (rc != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (rc);
}
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
rc = ena_change_mtu(adapter->ifp, adapter->ifp->if_mtu);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0))
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_refill_all_rx_bufs(adapter);
Rework counting of hardware statistics in ENA driver Do not read all statistics from the device, instead count them in the driver except from RX drops - they are received directly from the NIC in the AENQ descriptor. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: imp Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12852
2017-10-31 16:31:23 +00:00
ena_reset_counters((counter_u64_t *)&adapter->hw_stats,
sizeof(adapter->hw_stats));
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (0);
}
Split Rx/Tx from initialization code in ENA driver Move Rx/Tx routines to separate file. Some functions: * ena_restore_device, * ena_destroy_device, * ena_up, * ena_down, * ena_refill_rx_bufs could be reused in upcoming netmap code in the driver. To make it possible, they were moved to ena.h header. Differential Revision: https://reviews.freebsd.org/D21933 Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:44:26 +00:00
int
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_up(struct ena_adapter *adapter)
{
int rc = 0;
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(device_is_attached(adapter->pdev) == 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "device is not attached!\n");
return (ENXIO);
}
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (!ENA_FLAG_ISSET(ENA_FLAG_DEV_UP, adapter)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "device is going UP\n");
/* setup interrupts for IO queues */
Fix NULL pointer dereference in ena_up() If the call to ena_up() in ena_restore_device() fails, next usage of `ifconfig up` will cause NULL pointer dereference. This patch adds additional checks to prevent that. Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:42:52 +00:00
rc = ena_setup_io_intr(adapter);
if (unlikely(rc != 0)) {
ena_trace(ENA_ALERT, "error setting up IO interrupt\n");
goto error;
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rc = ena_request_io_irq(adapter);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
ena_trace(ENA_ALERT, "err_req_irq\n");
Fix NULL pointer dereference in ena_up() If the call to ena_up() in ena_restore_device() fails, next usage of `ifconfig up` will cause NULL pointer dereference. This patch adds additional checks to prevent that. Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:42:52 +00:00
goto error;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
/* allocate transmit descriptors */
rc = ena_setup_all_tx_resources(adapter);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
ena_trace(ENA_ALERT, "err_setup_tx\n");
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
goto err_setup_tx;
}
/* allocate receive descriptors */
rc = ena_setup_all_rx_resources(adapter);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
ena_trace(ENA_ALERT, "err_setup_rx\n");
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
goto err_setup_rx;
}
/* create IO queues for Rx & Tx */
rc = ena_create_io_queues(adapter);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_trace(ENA_ALERT,
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
"create IO queues failed\n");
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
goto err_io_que;
}
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (ENA_FLAG_ISSET(ENA_FLAG_LINK_UP, adapter))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
if_link_state_change(adapter->ifp, LINK_STATE_UP);
rc = ena_up_complete(adapter);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
goto err_up_complete;
counter_u64_add(adapter->dev_stats.interface_up, 1);
ena_update_hwassist(adapter);
if_setdrvflagbits(adapter->ifp, IFF_DRV_RUNNING,
IFF_DRV_OACTIVE);
Fix ENA keep-alive timeout due to prolonged reset When the ENA_FLAG_DEVICE_RUNNING flag is disabled, the AENQ handlers aren't executed. To fix that, the watchdog timestamp should be updated just before enabling the watchdog. Timer service was always being enabled, even if the device wasn't up before the reset. That shouldn't happen, as the timer service is being executed only for working interface. Differential Revision: https://reviews.freebsd.org/D21932 Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:39:54 +00:00
/* Activate timer service only if the device is running.
* If this flag is not set, it means that the driver is being
* reset and timer service will be activated afterwards.
*/
if (ENA_FLAG_ISSET(ENA_FLAG_DEVICE_RUNNING, adapter)) {
callout_reset_sbt(&adapter->timer_service, SBT_1S,
SBT_1S, ena_timer_service, (void *)adapter, 0);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ENA_FLAG_SET_ATOMIC(ENA_FLAG_DEV_UP, adapter);
Unmask all IO irqs after driver state is set as running If driver left MSI-x handlling routine because interface was put down, it is not unmasking IRQs, so any requesting interrupt will be awaiting for unmasking. On ena_up() routine all interrupts are being unmasked and any awaiting interrupt will be handled right away. If handler was executed before driver state was set as running, handling routine is being ended immediately, leaving IO irqs for given queue masked. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-04 00:02:28 +00:00
ena_unmask_all_io_irqs(adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
return (0);
err_up_complete:
ena_destroy_all_io_queues(adapter);
err_io_que:
ena_free_all_rx_resources(adapter);
err_setup_rx:
ena_free_all_tx_resources(adapter);
err_setup_tx:
ena_free_io_irq(adapter);
Fix NULL pointer dereference in ena_up() If the call to ena_up() in ena_restore_device() fails, next usage of `ifconfig up` will cause NULL pointer dereference. This patch adds additional checks to prevent that. Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:42:52 +00:00
error:
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (rc);
}
static uint64_t
ena_get_counter(if_t ifp, ift_counter cnt)
{
struct ena_adapter *adapter;
struct ena_hw_stats *stats;
adapter = if_getsoftc(ifp);
stats = &adapter->hw_stats;
switch (cnt) {
case IFCOUNTER_IPACKETS:
Rework counting of hardware statistics in ENA driver Do not read all statistics from the device, instead count them in the driver except from RX drops - they are received directly from the NIC in the AENQ descriptor. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: imp Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12852
2017-10-31 16:31:23 +00:00
return (counter_u64_fetch(stats->rx_packets));
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
case IFCOUNTER_OPACKETS:
Rework counting of hardware statistics in ENA driver Do not read all statistics from the device, instead count them in the driver except from RX drops - they are received directly from the NIC in the AENQ descriptor. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: imp Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12852
2017-10-31 16:31:23 +00:00
return (counter_u64_fetch(stats->tx_packets));
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
case IFCOUNTER_IBYTES:
Rework counting of hardware statistics in ENA driver Do not read all statistics from the device, instead count them in the driver except from RX drops - they are received directly from the NIC in the AENQ descriptor. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: imp Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12852
2017-10-31 16:31:23 +00:00
return (counter_u64_fetch(stats->rx_bytes));
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
case IFCOUNTER_OBYTES:
Rework counting of hardware statistics in ENA driver Do not read all statistics from the device, instead count them in the driver except from RX drops - they are received directly from the NIC in the AENQ descriptor. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: imp Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12852
2017-10-31 16:31:23 +00:00
return (counter_u64_fetch(stats->tx_bytes));
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
case IFCOUNTER_IQDROPS:
Rework counting of hardware statistics in ENA driver Do not read all statistics from the device, instead count them in the driver except from RX drops - they are received directly from the NIC in the AENQ descriptor. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: imp Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12852
2017-10-31 16:31:23 +00:00
return (counter_u64_fetch(stats->rx_drops));
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
default:
return (if_get_counter_default(ifp, cnt));
}
}
static int
ena_media_change(if_t ifp)
{
/* Media Change is not supported by firmware */
return (0);
}
static void
ena_media_status(if_t ifp, struct ifmediareq *ifmr)
{
struct ena_adapter *adapter = if_getsoftc(ifp);
Unify new line characters in the ENA driver Some messages were missing new line character and traces were not having unified behavior. To fix that, each trace and printout should add new line character at the end of each string - that should improve readability. Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:41:39 +00:00
ena_trace(ENA_DBG, "enter\n");
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Cleanup of the ENA driver header file Remove unused macros and fields - some of them were only initialized, without further usage. Implement minor style fixes and add required comments. On the occasion add missing TX completion counter, which was existing, but mistakenly remained unused. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12864
2017-11-09 12:07:02 +00:00
mtx_lock(&adapter->global_mtx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ifmr->ifm_status = IFM_AVALID;
ifmr->ifm_active = IFM_ETHER;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (!ENA_FLAG_ISSET(ENA_FLAG_LINK_UP, adapter)) {
Cleanup of the ENA driver header file Remove unused macros and fields - some of them were only initialized, without further usage. Implement minor style fixes and add required comments. On the occasion add missing TX completion counter, which was existing, but mistakenly remained unused. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12864
2017-11-09 12:07:02 +00:00
mtx_unlock(&adapter->global_mtx);
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ena_trace(ENA_INFO, "Link is down\n");
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return;
}
ifmr->ifm_status |= IFM_ACTIVE;
Do not specify active media type in ENA As the ENA is working only in virtualized environment, the active media is not specified. Instead, the active link type is set as unknown. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:06:07 +00:00
ifmr->ifm_active |= IFM_UNKNOWN | IFM_FDX;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Cleanup of the ENA driver header file Remove unused macros and fields - some of them were only initialized, without further usage. Implement minor style fixes and add required comments. On the occasion add missing TX completion counter, which was existing, but mistakenly remained unused. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12864
2017-11-09 12:07:02 +00:00
mtx_unlock(&adapter->global_mtx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
static void
ena_init(void *arg)
{
struct ena_adapter *adapter = (struct ena_adapter *)arg;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (!ENA_FLAG_ISSET(ENA_FLAG_DEV_UP, adapter)) {
Add missing lock upon initialization of the interface Lack of this lock was causing crash if down was called in parallel with the initialization routine. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-03 23:59:11 +00:00
sx_xlock(&adapter->ioctl_sx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_up(adapter);
Add missing lock upon initialization of the interface Lack of this lock was causing crash if down was called in parallel with the initialization routine. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-03 23:59:11 +00:00
sx_unlock(&adapter->ioctl_sx);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
static int
ena_ioctl(if_t ifp, u_long command, caddr_t data)
{
struct ena_adapter *adapter;
struct ifreq *ifr;
int rc;
adapter = ifp->if_softc;
ifr = (struct ifreq *)data;
/*
* Acquiring lock to prevent from running up and down routines parallel.
*/
rc = 0;
switch (command) {
case SIOCSIFMTU:
Skip setting the MTU for ENA if it is not changing On AWS, a network interface can get reinitialized every 30 minutes due to the MTU being (re)set when a new DHCP lease is obtained. This can cause packet drop, along with annoying syslog messages. Skip setting the MTU in the ena driver if the new MTU is the same as the old MTU. Note this fix is already in the netfront driver. Testing: Verified ena up/down messages do not appear every 30 min in /var/log/messages with the fix in place. Submitted by: Krishna Yenduri <kyenduri@brkt.com> Reviewed by: Michal Krawczyk <mk@semihalf.com>
2018-05-10 09:32:59 +00:00
if (ifp->if_mtu == ifr->ifr_mtu)
break;
Add locks before each ena_up and ena_down Lock only ena_up and ena_down calls in ioctl handler, instead of whole ioctl. Locking ioctl with sx lock that is sleepable, is not allowed in some cases, e.g. when multicast options are being changed. Additional locking was added in deatch function to prevent race condition with ioctl function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10924
2017-05-30 11:55:02 +00:00
sx_xlock(&adapter->ioctl_sx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_down(adapter);
ena_change_mtu(ifp, ifr->ifr_mtu);
rc = ena_up(adapter);
Add locks before each ena_up and ena_down Lock only ena_up and ena_down calls in ioctl handler, instead of whole ioctl. Locking ioctl with sx lock that is sleepable, is not allowed in some cases, e.g. when multicast options are being changed. Additional locking was added in deatch function to prevent race condition with ioctl function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10924
2017-05-30 11:55:02 +00:00
sx_unlock(&adapter->ioctl_sx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
break;
case SIOCSIFFLAGS:
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((ifp->if_flags & IFF_UP) != 0) {
if ((if_getdrvflags(ifp) & IFF_DRV_RUNNING) != 0) {
if ((ifp->if_flags & (IFF_PROMISC |
IFF_ALLMULTI)) != 0) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
"ioctl promisc/allmulti\n");
}
} else {
Add locks before each ena_up and ena_down Lock only ena_up and ena_down calls in ioctl handler, instead of whole ioctl. Locking ioctl with sx lock that is sleepable, is not allowed in some cases, e.g. when multicast options are being changed. Additional locking was added in deatch function to prevent race condition with ioctl function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10924
2017-05-30 11:55:02 +00:00
sx_xlock(&adapter->ioctl_sx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rc = ena_up(adapter);
Add locks before each ena_up and ena_down Lock only ena_up and ena_down calls in ioctl handler, instead of whole ioctl. Locking ioctl with sx lock that is sleepable, is not allowed in some cases, e.g. when multicast options are being changed. Additional locking was added in deatch function to prevent race condition with ioctl function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10924
2017-05-30 11:55:02 +00:00
sx_unlock(&adapter->ioctl_sx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
} else {
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((if_getdrvflags(ifp) & IFF_DRV_RUNNING) != 0) {
Add locks before each ena_up and ena_down Lock only ena_up and ena_down calls in ioctl handler, instead of whole ioctl. Locking ioctl with sx lock that is sleepable, is not allowed in some cases, e.g. when multicast options are being changed. Additional locking was added in deatch function to prevent race condition with ioctl function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10924
2017-05-30 11:55:02 +00:00
sx_xlock(&adapter->ioctl_sx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_down(adapter);
Add locks before each ena_up and ena_down Lock only ena_up and ena_down calls in ioctl handler, instead of whole ioctl. Locking ioctl with sx lock that is sleepable, is not allowed in some cases, e.g. when multicast options are being changed. Additional locking was added in deatch function to prevent race condition with ioctl function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10924
2017-05-30 11:55:02 +00:00
sx_unlock(&adapter->ioctl_sx);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
break;
case SIOCADDMULTI:
case SIOCDELMULTI:
break;
case SIOCSIFMEDIA:
case SIOCGIFMEDIA:
rc = ifmedia_ioctl(ifp, ifr, &adapter->media, command);
break;
case SIOCSIFCAP:
{
int reinit = 0;
if (ifr->ifr_reqcap != ifp->if_capenable) {
ifp->if_capenable = ifr->ifr_reqcap;
reinit = 1;
}
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((reinit != 0) &&
((if_getdrvflags(ifp) & IFF_DRV_RUNNING) != 0)) {
Add locks before each ena_up and ena_down Lock only ena_up and ena_down calls in ioctl handler, instead of whole ioctl. Locking ioctl with sx lock that is sleepable, is not allowed in some cases, e.g. when multicast options are being changed. Additional locking was added in deatch function to prevent race condition with ioctl function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10924
2017-05-30 11:55:02 +00:00
sx_xlock(&adapter->ioctl_sx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_down(adapter);
rc = ena_up(adapter);
Add locks before each ena_up and ena_down Lock only ena_up and ena_down calls in ioctl handler, instead of whole ioctl. Locking ioctl with sx lock that is sleepable, is not allowed in some cases, e.g. when multicast options are being changed. Additional locking was added in deatch function to prevent race condition with ioctl function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10924
2017-05-30 11:55:02 +00:00
sx_unlock(&adapter->ioctl_sx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
break;
default:
rc = ether_ioctl(ifp, command, data);
break;
}
return (rc);
}
static int
ena_get_dev_offloads(struct ena_com_dev_get_features_ctx *feat)
{
int caps = 0;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((feat->offload.tx &
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
(ENA_ADMIN_FEATURE_OFFLOAD_DESC_TX_L4_IPV4_CSUM_FULL_MASK |
ENA_ADMIN_FEATURE_OFFLOAD_DESC_TX_L4_IPV4_CSUM_PART_MASK |
Adjust ENA driver to the new ena-com Recent HAL change preparing to support ENAv2 required minor driver modifications. The ena_com_sq_empty_space() is not available in this ena-com, so it had to be replaced with ena_com_free_desc(). Moreover, the ena_com_admin_init() is no longer using 3rd argument indicating if the spin lock should be initialized, so it was removed. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:01:46 +00:00
ENA_ADMIN_FEATURE_OFFLOAD_DESC_TX_L3_CSUM_IPV4_MASK)) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
caps |= IFCAP_TXCSUM;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((feat->offload.tx &
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
(ENA_ADMIN_FEATURE_OFFLOAD_DESC_TX_L4_IPV6_CSUM_FULL_MASK |
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
ENA_ADMIN_FEATURE_OFFLOAD_DESC_TX_L4_IPV6_CSUM_PART_MASK)) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
caps |= IFCAP_TXCSUM_IPV6;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((feat->offload.tx &
ENA_ADMIN_FEATURE_OFFLOAD_DESC_TSO_IPV4_MASK) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
caps |= IFCAP_TSO4;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((feat->offload.tx &
ENA_ADMIN_FEATURE_OFFLOAD_DESC_TSO_IPV6_MASK) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
caps |= IFCAP_TSO6;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((feat->offload.rx_supported &
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
(ENA_ADMIN_FEATURE_OFFLOAD_DESC_RX_L4_IPV4_CSUM_MASK |
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
ENA_ADMIN_FEATURE_OFFLOAD_DESC_RX_L3_CSUM_IPV4_MASK)) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
caps |= IFCAP_RXCSUM;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((feat->offload.rx_supported &
ENA_ADMIN_FEATURE_OFFLOAD_DESC_RX_L4_IPV6_CSUM_MASK) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
caps |= IFCAP_RXCSUM_IPV6;
caps |= IFCAP_LRO | IFCAP_JUMBO_MTU;
return (caps);
}
static void
ena_update_host_info(struct ena_admin_host_info *host_info, if_t ifp)
{
host_info->supported_network_features[0] =
(uint32_t)if_getcapabilities(ifp);
}
static void
ena_update_hwassist(struct ena_adapter *adapter)
{
if_t ifp = adapter->ifp;
uint32_t feat = adapter->tx_offload_cap;
int cap = if_getcapenable(ifp);
int flags = 0;
if_clearhwassist(ifp);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((cap & IFCAP_TXCSUM) != 0) {
if ((feat &
ENA_ADMIN_FEATURE_OFFLOAD_DESC_TX_L3_CSUM_IPV4_MASK) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
flags |= CSUM_IP;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((feat &
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
(ENA_ADMIN_FEATURE_OFFLOAD_DESC_TX_L4_IPV4_CSUM_FULL_MASK |
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
ENA_ADMIN_FEATURE_OFFLOAD_DESC_TX_L4_IPV4_CSUM_PART_MASK)) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
flags |= CSUM_IP_UDP | CSUM_IP_TCP;
}
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((cap & IFCAP_TXCSUM_IPV6) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
flags |= CSUM_IP6_UDP | CSUM_IP6_TCP;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((cap & IFCAP_TSO4) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
flags |= CSUM_IP_TSO;
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if ((cap & IFCAP_TSO6) != 0)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
flags |= CSUM_IP6_TSO;
if_sethwassistbits(ifp, flags, 0);
}
static int
ena_setup_ifnet(device_t pdev, struct ena_adapter *adapter,
struct ena_com_dev_get_features_ctx *feat)
{
if_t ifp;
int caps = 0;
ifp = adapter->ifp = if_gethandle(IFT_ETHER);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(ifp == NULL)) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_ALERT, "can not allocate ifnet structure\n");
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (ENXIO);
}
if_initname(ifp, device_get_name(pdev), device_get_unit(pdev));
if_setdev(ifp, pdev);
if_setsoftc(ifp, adapter);
if_setflags(ifp, IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST);
if_setinitfn(ifp, ena_init);
if_settransmitfn(ifp, ena_mq_start);
if_setqflushfn(ifp, ena_qflush);
if_setioctlfn(ifp, ena_ioctl);
if_setgetcounterfn(ifp, ena_get_counter);
if_setsendqlen(ifp, adapter->tx_ring_size);
if_setsendqready(ifp);
if_setmtu(ifp, ETHERMTU);
if_setbaudrate(ifp, 0);
/* Zeroize capabilities... */
if_setcapabilities(ifp, 0);
if_setcapenable(ifp, 0);
/* check hardware support */
caps = ena_get_dev_offloads(feat);
/* ... and set them */
if_setcapabilitiesbit(ifp, caps, 0);
/* TSO parameters */
Fix creation of dma tags and TSO settings TSO settings were not reflecting real HW capabilities. DMA tags were created with wrong window - high address was the same as low, so excluding window was not working. Capabilities of TX dma transaction were not set properly - TSO max size had been increased and size of one segment had been adjusted. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc.
2017-07-04 00:08:47 +00:00
ifp->if_hw_tsomax = ENA_TSO_MAXSIZE -
(ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN);
ifp->if_hw_tsomaxsegcount = adapter->max_tx_sgl_size - 1;
ifp->if_hw_tsomaxsegsize = ENA_TSO_MAXSIZE;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
if_setifheaderlen(ifp, sizeof(struct ether_vlan_header));
if_setcapenable(ifp, if_getcapabilities(ifp));
/*
* Specify the media types supported by this adapter and register
* callbacks to update media and link information
*/
ifmedia_init(&adapter->media, IFM_IMASK,
ena_media_change, ena_media_status);
ifmedia_add(&adapter->media, IFM_ETHER | IFM_AUTO, 0, NULL);
ifmedia_set(&adapter->media, IFM_ETHER | IFM_AUTO);
ether_ifattach(ifp, adapter->mac_addr);
return (0);
}
Split Rx/Tx from initialization code in ENA driver Move Rx/Tx routines to separate file. Some functions: * ena_restore_device, * ena_destroy_device, * ena_up, * ena_down, * ena_refill_rx_bufs could be reused in upcoming netmap code in the driver. To make it possible, they were moved to ena.h header. Differential Revision: https://reviews.freebsd.org/D21933 Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:44:26 +00:00
void
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_down(struct ena_adapter *adapter)
{
Update ena-com HAL to v1.1.4.3 and update driver accordingly The newest ena-com HAL supports LLQv2 and introduces API changes. In order not to break the driver compilation it was updated/fixed in a following way: * Change version of the driver to 0.8.0 * Provide reset cause when triggering reset of the device * Reset device after attach fails * In the reset task free management irq after calling ena_down. Admin queue can still be used before ena_down is called, or when it is being handled * Do not reset device if ena_reset_task fails * Move call of the ena_com_dev_reset to the ena_down() routine - it should be called only if interface was up * Use different function for checking empty space on the sq ring (ena-com API change) * Fix typo on ENA_TX_CLEANUP_THRESHOLD * Change checking for EPERM with EOPNOTSUPP - change in the ena-com API * Minor style fixes Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Amazon.com, Inc. Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12143
2017-10-31 12:41:07 +00:00
int rc;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (ENA_FLAG_ISSET(ENA_FLAG_DEV_UP, adapter)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "device is going DOWN\n");
callout_drain(&adapter->timer_service);
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ENA_FLAG_CLEAR_ATOMIC(ENA_FLAG_DEV_UP, adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
if_setdrvflagbits(adapter->ifp, IFF_DRV_OACTIVE,
IFF_DRV_RUNNING);
ena_free_io_irq(adapter);
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET, adapter)) {
Update ena-com HAL to v1.1.4.3 and update driver accordingly The newest ena-com HAL supports LLQv2 and introduces API changes. In order not to break the driver compilation it was updated/fixed in a following way: * Change version of the driver to 0.8.0 * Provide reset cause when triggering reset of the device * Reset device after attach fails * In the reset task free management irq after calling ena_down. Admin queue can still be used before ena_down is called, or when it is being handled * Do not reset device if ena_reset_task fails * Move call of the ena_com_dev_reset to the ena_down() routine - it should be called only if interface was up * Use different function for checking empty space on the sq ring (ena-com API change) * Fix typo on ENA_TX_CLEANUP_THRESHOLD * Change checking for EPERM with EOPNOTSUPP - change in the ena-com API * Minor style fixes Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Amazon.com, Inc. Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12143
2017-10-31 12:41:07 +00:00
rc = ena_com_dev_reset(adapter->ena_dev,
adapter->reset_reason);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0))
Update ena-com HAL to v1.1.4.3 and update driver accordingly The newest ena-com HAL supports LLQv2 and introduces API changes. In order not to break the driver compilation it was updated/fixed in a following way: * Change version of the driver to 0.8.0 * Provide reset cause when triggering reset of the device * Reset device after attach fails * In the reset task free management irq after calling ena_down. Admin queue can still be used before ena_down is called, or when it is being handled * Do not reset device if ena_reset_task fails * Move call of the ena_com_dev_reset to the ena_down() routine - it should be called only if interface was up * Use different function for checking empty space on the sq ring (ena-com API change) * Fix typo on ENA_TX_CLEANUP_THRESHOLD * Change checking for EPERM with EOPNOTSUPP - change in the ena-com API * Minor style fixes Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Amazon.com, Inc. Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12143
2017-10-31 12:41:07 +00:00
device_printf(adapter->pdev,
"Device reset failed\n");
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_destroy_all_io_queues(adapter);
ena_free_all_tx_bufs(adapter);
ena_free_all_rx_bufs(adapter);
ena_free_all_tx_resources(adapter);
ena_free_all_rx_resources(adapter);
counter_u64_add(adapter->dev_stats.interface_down, 1);
}
}
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
static int
ena_calc_io_queue_num(struct ena_adapter *adapter,
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
struct ena_com_dev_get_features_ctx *get_feat_ctx)
{
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
struct ena_com_dev *ena_dev = adapter->ena_dev;
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
int io_tx_sq_num, io_tx_cq_num, io_rx_num, io_queue_num;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
/* Regular queues capabilities */
if (ena_dev->supported_features & BIT(ENA_ADMIN_MAX_QUEUES_EXT)) {
struct ena_admin_queue_ext_feature_fields *max_queue_ext =
&get_feat_ctx->max_queue_ext.max_queue_ext;
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
io_rx_num = min_t(int, max_queue_ext->max_rx_sq_num,
max_queue_ext->max_rx_cq_num);
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
io_tx_sq_num = max_queue_ext->max_tx_sq_num;
io_tx_cq_num = max_queue_ext->max_tx_cq_num;
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
} else {
struct ena_admin_queue_feature_desc *max_queues =
&get_feat_ctx->max_queues;
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
io_tx_sq_num = max_queues->max_sq_num;
io_tx_cq_num = max_queues->max_cq_num;
io_rx_num = min_t(int, io_tx_sq_num, io_tx_cq_num);
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
/* In case of LLQ use the llq fields for the tx SQ/CQ */
if (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_DEV)
io_tx_sq_num = get_feat_ctx->llq.max_llq_num;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
io_queue_num = min_t(int, mp_ncpus, ENA_MAX_NUM_IO_QUEUES);
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
io_queue_num = min_t(int, io_queue_num, io_rx_num);
io_queue_num = min_t(int, io_queue_num, io_tx_sq_num);
io_queue_num = min_t(int, io_queue_num, io_tx_cq_num);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* 1 IRQ for for mgmnt and 1 IRQ for each TX/RX pair */
io_queue_num = min_t(int, io_queue_num,
pci_msix_count(adapter->pdev) - 1);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
return (io_queue_num);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
static int
ena_enable_wc(struct resource *res)
{
Add WC support for arm64 in the ENA driver As the pmamp_change_attr() is public on arm64 since r351131, it can be used on the arm64 to map memory range as with the write combined attribute. It requires the driver to use generic VM_MEMATTR_WRITE_COMBINING flag instead of the x86 specific PAT_WRITE_COMBINING. Differential Revision: https://reviews.freebsd.org/D21931 Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:38:17 +00:00
#if defined(__i386) || defined(__amd64) || defined(__aarch64__)
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
vm_offset_t va;
vm_size_t len;
int rc;
va = (vm_offset_t)rman_get_virtual(res);
len = rman_get_size(res);
/* Enable write combining */
Add WC support for arm64 in the ENA driver As the pmamp_change_attr() is public on arm64 since r351131, it can be used on the arm64 to map memory range as with the write combined attribute. It requires the driver to use generic VM_MEMATTR_WRITE_COMBINING flag instead of the x86 specific PAT_WRITE_COMBINING. Differential Revision: https://reviews.freebsd.org/D21931 Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:38:17 +00:00
rc = pmap_change_attr(va, len, VM_MEMATTR_WRITE_COMBINING);
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
if (unlikely(rc != 0)) {
ena_trace(ENA_ALERT, "pmap_change_attr failed, %d\n", rc);
return (rc);
}
return (0);
#endif
return (EOPNOTSUPP);
}
static int
ena_set_queues_placement_policy(device_t pdev, struct ena_com_dev *ena_dev,
struct ena_admin_feature_llq_desc *llq,
struct ena_llq_configurations *llq_default_configurations)
{
struct ena_adapter *adapter = device_get_softc(pdev);
int rc, rid;
uint32_t llq_feature_mask;
llq_feature_mask = 1 << ENA_ADMIN_LLQ;
if (!(ena_dev->supported_features & llq_feature_mask)) {
device_printf(pdev,
"LLQ is not supported. Fallback to host mode policy.\n");
ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST;
return (0);
}
rc = ena_com_config_dev_mode(ena_dev, llq, llq_default_configurations);
if (unlikely(rc != 0)) {
device_printf(pdev, "Failed to configure the device mode. "
"Fallback to host mode policy.\n");
ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST;
return (0);
}
/* Nothing to config, exit */
if (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_HOST)
return (0);
/* Try to allocate resources for LLQ bar */
rid = PCIR_BAR(ENA_MEM_BAR);
adapter->memory = bus_alloc_resource_any(pdev, SYS_RES_MEMORY,
&rid, RF_ACTIVE);
if (unlikely(adapter->memory == NULL)) {
device_printf(pdev, "unable to allocate LLQ bar resource. "
"Fallback to host mode policy.\n");
ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST;
return (0);
}
/* Enable write combining for better LLQ performance */
rc = ena_enable_wc(adapter->memory);
if (unlikely(rc != 0)) {
device_printf(pdev, "failed to enable write combining.\n");
return (rc);
}
/*
* Save virtual address of the device's memory region
* for the ena_com layer.
*/
ena_dev->mem_bar = rman_get_virtual(adapter->memory);
return (0);
}
static inline
void set_default_llq_configurations(struct ena_llq_configurations *llq_config)
{
llq_config->llq_header_location = ENA_ADMIN_INLINE_HEADER;
llq_config->llq_ring_entry_size = ENA_ADMIN_LIST_ENTRY_SIZE_128B;
llq_config->llq_stride_ctrl = ENA_ADMIN_MULTIPLE_DESCS_PER_ENTRY;
llq_config->llq_num_decs_before_header =
ENA_ADMIN_LLQ_NUM_DESCS_BEFORE_HEADER_2;
llq_config->llq_ring_entry_size_value = 128;
}
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
static int
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
ena_calc_queue_size(struct ena_adapter *adapter,
struct ena_calc_queue_size_ctx *ctx)
{
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
struct ena_admin_feature_llq_desc *llq = &ctx->get_feat_ctx->llq;
struct ena_com_dev *ena_dev = ctx->ena_dev;
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
uint32_t tx_queue_size = ENA_DEFAULT_RING_SIZE;
uint32_t rx_queue_size = adapter->rx_ring_size;
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
if (ena_dev->supported_features & BIT(ENA_ADMIN_MAX_QUEUES_EXT)) {
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
struct ena_admin_queue_ext_feature_fields *max_queue_ext =
&ctx->get_feat_ctx->max_queue_ext.max_queue_ext;
rx_queue_size = min_t(uint32_t, rx_queue_size,
max_queue_ext->max_rx_cq_depth);
rx_queue_size = min_t(uint32_t, rx_queue_size,
max_queue_ext->max_rx_sq_depth);
tx_queue_size = min_t(uint32_t, tx_queue_size,
max_queue_ext->max_tx_cq_depth);
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
if (ena_dev->tx_mem_queue_type ==
ENA_ADMIN_PLACEMENT_POLICY_DEV)
tx_queue_size = min_t(uint32_t, tx_queue_size,
llq->max_llq_depth);
else
tx_queue_size = min_t(uint32_t, tx_queue_size,
max_queue_ext->max_tx_sq_depth);
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
ctx->max_rx_sgl_size = min_t(uint16_t, ENA_PKT_MAX_BUFS,
max_queue_ext->max_per_packet_rx_descs);
ctx->max_tx_sgl_size = min_t(uint16_t, ENA_PKT_MAX_BUFS,
max_queue_ext->max_per_packet_tx_descs);
} else {
struct ena_admin_queue_feature_desc *max_queues =
&ctx->get_feat_ctx->max_queues;
rx_queue_size = min_t(uint32_t, rx_queue_size,
max_queues->max_cq_depth);
rx_queue_size = min_t(uint32_t, rx_queue_size,
max_queues->max_sq_depth);
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
tx_queue_size = min_t(uint32_t, tx_queue_size,
max_queues->max_cq_depth);
if (ena_dev->tx_mem_queue_type ==
ENA_ADMIN_PLACEMENT_POLICY_DEV)
tx_queue_size = min_t(uint32_t, tx_queue_size,
llq->max_llq_depth);
else
tx_queue_size = min_t(uint32_t, tx_queue_size,
max_queues->max_sq_depth);
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
ctx->max_rx_sgl_size = min_t(uint16_t, ENA_PKT_MAX_BUFS,
max_queues->max_packet_tx_descs);
ctx->max_tx_sgl_size = min_t(uint16_t, ENA_PKT_MAX_BUFS,
max_queues->max_packet_rx_descs);
}
/* round down to the nearest power of 2 */
rx_queue_size = 1 << (fls(rx_queue_size) - 1);
tx_queue_size = 1 << (fls(tx_queue_size) - 1);
if (unlikely(rx_queue_size == 0 || tx_queue_size == 0)) {
device_printf(ctx->pdev, "Invalid queue size\n");
return (EFAULT);
}
ctx->rx_queue_size = rx_queue_size;
ctx->tx_queue_size = tx_queue_size;
return (0);
}
static int
ena_handle_updated_queues(struct ena_adapter *adapter,
struct ena_com_dev_get_features_ctx *get_feat_ctx)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
{
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
struct ena_com_dev *ena_dev = adapter->ena_dev;
struct ena_calc_queue_size_ctx calc_queue_ctx = { 0 };
device_t pdev = adapter->pdev;
bool are_queues_changed = false;
int io_queue_num, rc;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
calc_queue_ctx.ena_dev = ena_dev;
calc_queue_ctx.get_feat_ctx = get_feat_ctx;
calc_queue_ctx.pdev = pdev;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
io_queue_num = ena_calc_io_queue_num(adapter, get_feat_ctx);
rc = ena_calc_queue_size(adapter, &calc_queue_ctx);
if (unlikely(rc != 0 || io_queue_num <= 0))
return EFAULT;
if (adapter->tx_ring->buf_ring_size != adapter->buf_ring_size)
are_queues_changed = true;
if (unlikely(adapter->tx_ring_size > calc_queue_ctx.tx_queue_size ||
adapter->rx_ring_size > calc_queue_ctx.rx_queue_size)) {
device_printf(pdev,
"Not enough resources to allocate requested queue sizes "
"(TX,RX)=(%d,%d), falling back to queue sizes "
"(TX,RX)=(%d,%d)\n",
adapter->tx_ring_size,
adapter->rx_ring_size,
calc_queue_ctx.tx_queue_size,
calc_queue_ctx.rx_queue_size);
adapter->tx_ring_size = calc_queue_ctx.tx_queue_size;
adapter->rx_ring_size = calc_queue_ctx.rx_queue_size;
adapter->max_tx_sgl_size = calc_queue_ctx.max_tx_sgl_size;
adapter->max_rx_sgl_size = calc_queue_ctx.max_rx_sgl_size;
are_queues_changed = true;
}
if (unlikely(adapter->num_queues > io_queue_num)) {
device_printf(pdev,
"Not enough resources to allocate %d queues, "
"falling back to %d queues\n",
adapter->num_queues, io_queue_num);
adapter->num_queues = io_queue_num;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (ENA_FLAG_ISSET(ENA_FLAG_RSS_ACTIVE, adapter)) {
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
ena_com_rss_destroy(ena_dev);
rc = ena_rss_init_default(adapter);
if (unlikely(rc != 0) && (rc != EOPNOTSUPP)) {
device_printf(pdev, "Cannot init RSS rc: %d\n",
rc);
return (rc);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
are_queues_changed = true;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
if (unlikely(are_queues_changed)) {
ena_free_all_io_rings_resources(adapter);
ena_init_io_rings(adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
return (0);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
static int
ena_rss_init_default(struct ena_adapter *adapter)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
{
struct ena_com_dev *ena_dev = adapter->ena_dev;
device_t dev = adapter->pdev;
int qid, rc, i;
rc = ena_com_rss_init(ena_dev, ENA_RX_RSS_TABLE_LOG_SIZE);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (unlikely(rc != 0)) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
device_printf(dev, "Cannot init indirect table\n");
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
for (i = 0; i < ENA_RX_RSS_TABLE_SIZE; i++) {
qid = i % adapter->num_queues;
rc = ena_com_indirect_table_fill_entry(ena_dev, i,
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
ENA_IO_RXQ_IDX(qid));
if (unlikely((rc != 0) && (rc != EOPNOTSUPP))) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(dev, "Cannot fill indirect table\n");
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
goto err_rss_destroy;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
rc = ena_com_fill_hash_function(ena_dev, ENA_ADMIN_CRC32, NULL,
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
ENA_HASH_KEY_SIZE, 0xFFFFFFFF);
if (unlikely((rc != 0) && (rc != EOPNOTSUPP))) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(dev, "Cannot fill hash function\n");
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
goto err_rss_destroy;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
rc = ena_com_set_default_hash_ctrl(ena_dev);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (unlikely((rc != 0) && (rc != EOPNOTSUPP))) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(dev, "Cannot fill hash control\n");
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
goto err_rss_destroy;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
return (0);
Fix error handling in the ENA driver and lock drbr_free() call Some goto tags were renamed for consistency, and few error handling routines were reworked. The drbr_free() must be locked just in case code will change in the future - for now, it should never be an issue, because drbr is being flushed in the ena_down() call, and the lock is required only when there are some mbufs inside. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12859
2017-11-09 11:54:32 +00:00
err_rss_destroy:
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_com_rss_destroy(ena_dev);
return (rc);
}
static void
ena_rss_init_default_deferred(void *arg)
{
struct ena_adapter *adapter;
devclass_t dc;
int max;
int rc;
dc = devclass_find("ena");
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(dc == NULL)) {
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
ena_trace(ENA_ALERT, "No devclass ena\n");
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return;
}
max = devclass_get_maxunit(dc);
while (max-- >= 0) {
adapter = devclass_get_softc(dc, max);
if (adapter != NULL) {
rc = ena_rss_init_default(adapter);
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ENA_FLAG_SET_ATOMIC(ENA_FLAG_RSS_ACTIVE, adapter);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
"WARNING: RSS was not properly initialized,"
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
" it will affect bandwidth\n");
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ENA_FLAG_CLEAR_ATOMIC(ENA_FLAG_RSS_ACTIVE, adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
}
}
SYSINIT(ena_rss_init, SI_SUB_KICK_SCHEDULER, SI_ORDER_SECOND, ena_rss_init_default_deferred, NULL);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
static void
Fill bdf field of the host_info structure in ENA The host info bdf field is the abbreviation for the bus, device, function of the PCI on which the device is being attached to. Now the driver is filling information about that using FreeBSD RID resource. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:35:02 +00:00
ena_config_host_info(struct ena_com_dev *ena_dev, device_t dev)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
{
struct ena_admin_host_info *host_info;
Fill bdf field of the host_info structure in ENA The host info bdf field is the abbreviation for the bus, device, function of the PCI on which the device is being attached to. Now the driver is filling information about that using FreeBSD RID resource. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:35:02 +00:00
uintptr_t rid;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
int rc;
/* Allocate only the host info */
rc = ena_com_allocate_host_info(ena_dev);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_trace(ENA_ALERT, "Cannot allocate host info\n");
return;
}
host_info = ena_dev->host_attr.host_info;
Fill bdf field of the host_info structure in ENA The host info bdf field is the abbreviation for the bus, device, function of the PCI on which the device is being attached to. Now the driver is filling information about that using FreeBSD RID resource. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:35:02 +00:00
if (pci_get_id(dev, PCI_ID_RID, &rid) == 0)
host_info->bdf = rid;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
host_info->os_type = ENA_ADMIN_OS_FREEBSD;
host_info->kernel_ver = osreldate;
sprintf(host_info->kernel_ver_str, "%d", osreldate);
host_info->os_dist = 0;
strncpy(host_info->os_dist_str, osrelease,
sizeof(host_info->os_dist_str) - 1);
host_info->driver_version =
(DRV_MODULE_VER_MAJOR) |
(DRV_MODULE_VER_MINOR << ENA_ADMIN_HOST_INFO_MINOR_SHIFT) |
(DRV_MODULE_VER_SUBMINOR << ENA_ADMIN_HOST_INFO_SUB_MINOR_SHIFT);
Fill number of CPUs field on ENA host_info structure The new ena_com allows the number of CPUs to be passed to the device in the host info structure as a hint. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:15:38 +00:00
host_info->num_cpus = mp_ncpus;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rc = ena_com_set_host_attributes(ena_dev);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Update ena-com HAL to v1.1.4.3 and update driver accordingly The newest ena-com HAL supports LLQv2 and introduces API changes. In order not to break the driver compilation it was updated/fixed in a following way: * Change version of the driver to 0.8.0 * Provide reset cause when triggering reset of the device * Reset device after attach fails * In the reset task free management irq after calling ena_down. Admin queue can still be used before ena_down is called, or when it is being handled * Do not reset device if ena_reset_task fails * Move call of the ena_com_dev_reset to the ena_down() routine - it should be called only if interface was up * Use different function for checking empty space on the sq ring (ena-com API change) * Fix typo on ENA_TX_CLEANUP_THRESHOLD * Change checking for EPERM with EOPNOTSUPP - change in the ena-com API * Minor style fixes Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Amazon.com, Inc. Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12143
2017-10-31 12:41:07 +00:00
if (rc == EOPNOTSUPP)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_trace(ENA_WARNING, "Cannot set host attributes\n");
else
ena_trace(ENA_ALERT, "Cannot set host attributes\n");
goto err;
}
return;
err:
ena_com_delete_host_info(ena_dev);
}
static int
ena_device_init(struct ena_adapter *adapter, device_t pdev,
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
struct ena_com_dev_get_features_ctx *get_feat_ctx, int *wd_active)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
{
struct ena_com_dev* ena_dev = adapter->ena_dev;
bool readless_supported;
uint32_t aenq_groups;
int dma_width;
int rc;
rc = ena_com_mmio_reg_read_request_init(ena_dev);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(pdev, "failed to init mmio read less\n");
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
/*
* The PCIe configuration space revision id indicate if mmio reg
* read is disabled
*/
readless_supported = !(pci_get_revid(pdev) & ENA_MMIO_DISABLE_REG_READ);
ena_com_set_mmio_read_mode(ena_dev, readless_supported);
Update ena-com HAL to v1.1.4.3 and update driver accordingly The newest ena-com HAL supports LLQv2 and introduces API changes. In order not to break the driver compilation it was updated/fixed in a following way: * Change version of the driver to 0.8.0 * Provide reset cause when triggering reset of the device * Reset device after attach fails * In the reset task free management irq after calling ena_down. Admin queue can still be used before ena_down is called, or when it is being handled * Do not reset device if ena_reset_task fails * Move call of the ena_com_dev_reset to the ena_down() routine - it should be called only if interface was up * Use different function for checking empty space on the sq ring (ena-com API change) * Fix typo on ENA_TX_CLEANUP_THRESHOLD * Change checking for EPERM with EOPNOTSUPP - change in the ena-com API * Minor style fixes Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Amazon.com, Inc. Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12143
2017-10-31 12:41:07 +00:00
rc = ena_com_dev_reset(ena_dev, ENA_REGS_RESET_NORMAL);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(pdev, "Can not reset device\n");
goto err_mmio_read_less;
}
rc = ena_com_validate_version(ena_dev);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(pdev, "device version is too low\n");
goto err_mmio_read_less;
}
dma_width = ena_com_get_dma_width(ena_dev);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(dma_width < 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(pdev, "Invalid dma width value %d", dma_width);
rc = dma_width;
goto err_mmio_read_less;
}
adapter->dma_width = dma_width;
/* ENA admin level init */
Adjust ENA driver to the new ena-com Recent HAL change preparing to support ENAv2 required minor driver modifications. The ena_com_sq_empty_space() is not available in this ena-com, so it had to be replaced with ena_com_free_desc(). Moreover, the ena_com_admin_init() is no longer using 3rd argument indicating if the spin lock should be initialized, so it was removed. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:01:46 +00:00
rc = ena_com_admin_init(ena_dev, &aenq_handlers);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(pdev,
"Can not initialize ena admin queue with device\n");
goto err_mmio_read_less;
}
/*
* To enable the msix interrupts the driver needs to know the number
* of queues. So the driver uses polling mode to retrieve this
* information
*/
ena_com_set_admin_polling_mode(ena_dev, true);
Fill bdf field of the host_info structure in ENA The host info bdf field is the abbreviation for the bus, device, function of the PCI on which the device is being attached to. Now the driver is filling information about that using FreeBSD RID resource. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:35:02 +00:00
ena_config_host_info(ena_dev, pdev);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Get Device Attributes */
rc = ena_com_get_dev_attr_feat(ena_dev, get_feat_ctx);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(pdev,
"Cannot get attribute for ena device rc: %d\n", rc);
goto err_admin_init;
}
Print information when ENA admin error occurs ENA_ADMIN_FATAL_ERROR and ENA_ADMIN_WARNING aenq groups were indicated as supported, so the unimplemented_aenq_handler() will print out error message, whenever an error will occur within the ENA admin context. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:08:00 +00:00
aenq_groups = BIT(ENA_ADMIN_LINK_CHANGE) |
BIT(ENA_ADMIN_FATAL_ERROR) |
BIT(ENA_ADMIN_WARNING) |
Add notification AENQ handler for ENA Notification AENQ handler is responsible for handling requests from ENA device. Missing Tx threshold, Tx timeout and keep alive timeout can be set using hints from the aenq descriptor which can be delivered in the ENA admin notification. The queue suspending and resuming tasks are not supported by the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:09:53 +00:00
BIT(ENA_ADMIN_NOTIFICATION) |
Print information when ENA admin error occurs ENA_ADMIN_FATAL_ERROR and ENA_ADMIN_WARNING aenq groups were indicated as supported, so the unimplemented_aenq_handler() will print out error message, whenever an error will occur within the ENA admin context. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:08:00 +00:00
BIT(ENA_ADMIN_KEEP_ALIVE);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
aenq_groups &= get_feat_ctx->aenq.supported_groups;
rc = ena_com_set_aenq_config(ena_dev, aenq_groups);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(pdev, "Cannot configure aenq groups rc: %d\n", rc);
goto err_admin_init;
}
*wd_active = !!(aenq_groups & BIT(ENA_ADMIN_KEEP_ALIVE));
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
return (0);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
err_admin_init:
ena_com_delete_host_info(ena_dev);
ena_com_admin_destroy(ena_dev);
err_mmio_read_less:
ena_com_mmio_reg_read_request_destroy(ena_dev);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
static int ena_enable_msix_and_set_admin_interrupts(struct ena_adapter *adapter,
int io_vectors)
{
struct ena_com_dev *ena_dev = adapter->ena_dev;
int rc;
rc = ena_enable_msix(adapter);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "Error with MSI-X enablement\n");
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
ena_setup_mgmnt_intr(adapter);
rc = ena_request_mgmnt_irq(adapter);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "Cannot setup mgmnt queue intr\n");
goto err_disable_msix;
}
ena_com_set_admin_polling_mode(ena_dev, false);
ena_com_admin_aenq_enable(ena_dev);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
return (0);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
err_disable_msix:
ena_disable_msix(adapter);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
/* Function called on ENA_ADMIN_KEEP_ALIVE event */
static void ena_keep_alive_wd(void *adapter_data,
struct ena_admin_aenq_entry *aenq_e)
{
struct ena_adapter *adapter = (struct ena_adapter *)adapter_data;
Rework counting of hardware statistics in ENA driver Do not read all statistics from the device, instead count them in the driver except from RX drops - they are received directly from the NIC in the AENQ descriptor. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: imp Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12852
2017-10-31 16:31:23 +00:00
struct ena_admin_aenq_keep_alive_desc *desc;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
sbintime_t stime;
Rework counting of hardware statistics in ENA driver Do not read all statistics from the device, instead count them in the driver except from RX drops - they are received directly from the NIC in the AENQ descriptor. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: imp Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12852
2017-10-31 16:31:23 +00:00
uint64_t rx_drops;
desc = (struct ena_admin_aenq_keep_alive_desc *)aenq_e;
rx_drops = ((uint64_t)desc->rx_drops_high << 32) | desc->rx_drops_low;
counter_u64_zero(adapter->hw_stats.rx_drops);
counter_u64_add(adapter->hw_stats.rx_drops, rx_drops);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
stime = getsbinuptime();
atomic_store_rel_64(&adapter->keep_alive_timestamp, stime);
}
/* Check for keep alive expiration */
static void check_for_missing_keep_alive(struct ena_adapter *adapter)
{
sbintime_t timestamp, time;
if (adapter->wd_active == 0)
return;
Add notification AENQ handler for ENA Notification AENQ handler is responsible for handling requests from ENA device. Missing Tx threshold, Tx timeout and keep alive timeout can be set using hints from the aenq descriptor which can be delivered in the ENA admin notification. The queue suspending and resuming tasks are not supported by the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:09:53 +00:00
if (adapter->keep_alive_timeout == ENA_HW_HINTS_NO_TIMEOUT)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return;
timestamp = atomic_load_acq_64(&adapter->keep_alive_timestamp);
time = getsbinuptime() - timestamp;
if (unlikely(time > adapter->keep_alive_timeout)) {
device_printf(adapter->pdev,
"Keep alive watchdog timeout.\n");
counter_u64_add(adapter->dev_stats.wd_expired, 1);
Improve ENA reset handling For easier debugging, the reset is being triggered and the reset reason is being set only in case it is done for the first time. Such approach will ensure that the first reset reason is not going to be overwritten and will make it easier for debugging. Also, add a reset trigger upon invalid Tx requested ID. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:45:41 +00:00
if (likely(!ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET, adapter))) {
adapter->reset_reason = ENA_REGS_RESET_KEEP_ALIVE_TO;
ENA_FLAG_SET_ATOMIC(ENA_FLAG_TRIGGER_RESET, adapter);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
/* Check if admin queue is enabled */
static void check_for_admin_com_state(struct ena_adapter *adapter)
{
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (unlikely(ena_com_get_admin_running_state(adapter->ena_dev) ==
false)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
"ENA admin queue is not in running state!\n");
counter_u64_add(adapter->dev_stats.admin_q_pause, 1);
Improve ENA reset handling For easier debugging, the reset is being triggered and the reset reason is being set only in case it is done for the first time. Such approach will ensure that the first reset reason is not going to be overwritten and will make it easier for debugging. Also, add a reset trigger upon invalid Tx requested ID. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:45:41 +00:00
if (likely(!ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET, adapter))) {
adapter->reset_reason = ENA_REGS_RESET_ADMIN_TO;
ENA_FLAG_SET_ATOMIC(ENA_FLAG_TRIGGER_RESET, adapter);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
Split function checking for missing TX completion in ENA driver Pure cosmetic change for better readability of the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12857
2017-11-09 11:50:52 +00:00
static int
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
check_for_rx_interrupt_queue(struct ena_adapter *adapter,
struct ena_ring *rx_ring)
{
if (likely(rx_ring->first_interrupt))
return (0);
if (ena_com_cq_empty(rx_ring->ena_com_io_cq))
return (0);
rx_ring->no_interrupt_event_cnt++;
if (rx_ring->no_interrupt_event_cnt == ENA_MAX_NO_INTERRUPT_ITERATIONS) {
device_printf(adapter->pdev, "Potential MSIX issue on Rx side "
Improve ENA reset handling For easier debugging, the reset is being triggered and the reset reason is being set only in case it is done for the first time. Such approach will ensure that the first reset reason is not going to be overwritten and will make it easier for debugging. Also, add a reset trigger upon invalid Tx requested ID. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:45:41 +00:00
"Queue = %d. Reset the device\n", rx_ring->qid);
if (likely(!ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET, adapter))) {
adapter->reset_reason = ENA_REGS_RESET_MISS_INTERRUPT;
ENA_FLAG_SET_ATOMIC(ENA_FLAG_TRIGGER_RESET, adapter);
}
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
return (EIO);
}
return (0);
}
static int
check_missing_comp_in_tx_queue(struct ena_adapter *adapter,
Split function checking for missing TX completion in ENA driver Pure cosmetic change for better readability of the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12857
2017-11-09 11:50:52 +00:00
struct ena_ring *tx_ring)
{
struct bintime curtime, time;
struct ena_tx_buffer *tx_buf;
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
sbintime_t time_offset;
Split function checking for missing TX completion in ENA driver Pure cosmetic change for better readability of the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12857
2017-11-09 11:50:52 +00:00
uint32_t missed_tx = 0;
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
int i, rc = 0;
Split function checking for missing TX completion in ENA driver Pure cosmetic change for better readability of the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12857
2017-11-09 11:50:52 +00:00
getbinuptime(&curtime);
for (i = 0; i < tx_ring->ring_size; i++) {
tx_buf = &tx_ring->tx_buffer_info[i];
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (bintime_isset(&tx_buf->timestamp) == 0)
Split function checking for missing TX completion in ENA driver Pure cosmetic change for better readability of the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12857
2017-11-09 11:50:52 +00:00
continue;
time = curtime;
bintime_sub(&time, &tx_buf->timestamp);
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
time_offset = bttosbt(time);
if (unlikely(!tx_ring->first_interrupt &&
time_offset > 2 * adapter->missing_tx_timeout)) {
/*
* If after graceful period interrupt is still not
* received, we schedule a reset.
*/
device_printf(adapter->pdev,
"Potential MSIX issue on Tx side Queue = %d. "
"Reset the device\n", tx_ring->qid);
Improve ENA reset handling For easier debugging, the reset is being triggered and the reset reason is being set only in case it is done for the first time. Such approach will ensure that the first reset reason is not going to be overwritten and will make it easier for debugging. Also, add a reset trigger upon invalid Tx requested ID. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:45:41 +00:00
if (likely(!ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET,
adapter))) {
adapter->reset_reason =
ENA_REGS_RESET_MISS_INTERRUPT;
ENA_FLAG_SET_ATOMIC(ENA_FLAG_TRIGGER_RESET,
adapter);
}
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
return (EIO);
}
Split function checking for missing TX completion in ENA driver Pure cosmetic change for better readability of the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12857
2017-11-09 11:50:52 +00:00
/* Check again if packet is still waiting */
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
if (unlikely(time_offset > adapter->missing_tx_timeout)) {
Split function checking for missing TX completion in ENA driver Pure cosmetic change for better readability of the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12857
2017-11-09 11:50:52 +00:00
if (!tx_buf->print_once)
ena_trace(ENA_WARNING, "Found a Tx that wasn't "
"completed on time, qid %d, index %d.\n",
tx_ring->qid, i);
tx_buf->print_once = true;
missed_tx++;
}
}
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
if (unlikely(missed_tx > adapter->missing_tx_threshold)) {
device_printf(adapter->pdev,
"The number of lost tx completion is above the threshold "
"(%d > %d). Reset the device\n",
missed_tx, adapter->missing_tx_threshold);
Improve ENA reset handling For easier debugging, the reset is being triggered and the reset reason is being set only in case it is done for the first time. Such approach will ensure that the first reset reason is not going to be overwritten and will make it easier for debugging. Also, add a reset trigger upon invalid Tx requested ID. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:45:41 +00:00
if (likely(!ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET, adapter))) {
adapter->reset_reason = ENA_REGS_RESET_MISS_TX_CMPL;
ENA_FLAG_SET_ATOMIC(ENA_FLAG_TRIGGER_RESET, adapter);
}
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
rc = EIO;
}
counter_u64_add(tx_ring->tx_stats.missing_tx_comp, missed_tx);
return (rc);
Split function checking for missing TX completion in ENA driver Pure cosmetic change for better readability of the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12857
2017-11-09 11:50:52 +00:00
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/*
* Check for TX which were not completed on time.
* Timeout is defined by "missing_tx_timeout".
* Reset will be performed if number of incompleted
* transactions exceeds "missing_tx_threshold".
*/
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
static void
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
check_for_missing_completions(struct ena_adapter *adapter)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
{
struct ena_ring *tx_ring;
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
struct ena_ring *rx_ring;
Split function checking for missing TX completion in ENA driver Pure cosmetic change for better readability of the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12857
2017-11-09 11:50:52 +00:00
int i, budget, rc;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Make sure the driver doesn't turn the device in other process */
rmb();
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (!ENA_FLAG_ISSET(ENA_FLAG_DEV_UP, adapter))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET, adapter))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return;
Add notification AENQ handler for ENA Notification AENQ handler is responsible for handling requests from ENA device. Missing Tx threshold, Tx timeout and keep alive timeout can be set using hints from the aenq descriptor which can be delivered in the ENA admin notification. The queue suspending and resuming tasks are not supported by the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:09:53 +00:00
if (adapter->missing_tx_timeout == ENA_HW_HINTS_NO_TIMEOUT)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return;
budget = adapter->missing_tx_max_queues;
for (i = adapter->next_monitored_tx_qid; i < adapter->num_queues; i++) {
tx_ring = &adapter->tx_ring[i];
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
rx_ring = &adapter->rx_ring[i];
rc = check_missing_comp_in_tx_queue(adapter, tx_ring);
if (unlikely(rc != 0))
return;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
rc = check_for_rx_interrupt_queue(adapter, rx_ring);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (unlikely(rc != 0))
Split function checking for missing TX completion in ENA driver Pure cosmetic change for better readability of the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12857
2017-11-09 11:50:52 +00:00
return;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
budget--;
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
if (budget == 0) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
i++;
break;
}
}
adapter->next_monitored_tx_qid = i % adapter->num_queues;
}
Lock optimization in ENA Handle IO interrupts using filter routine. That way, the main cleanup task could be moved to the separate thread using taskqueue. The deferred Rx cleanup task was removed, and now the cleanup task is begin called instead. That way, the Rx lock could be removed. In addition, Queue management (wake up and stop TX ring) was added, so the TX cleanup task can be performed mostly lockless. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:29:24 +00:00
/* trigger rx cleanup after 2 consecutive detections */
Check for Rx ring state to prevent from stall in the ENA driver In case when Rx ring is full and driver will fail to allocate Rx mbufs, the ring could be stalled. Keep alive is checking every second for Rx ring state, and if it is full for two cycles, then trigger rx_cleanup routine in another thread. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12856
2017-11-09 11:48:22 +00:00
#define EMPTY_RX_REFILL 2
/* For the rare case where the device runs out of Rx descriptors and the
* msix handler failed to refill new Rx descriptors (due to a lack of memory
* for example).
* This case will lead to a deadlock:
* The device won't send interrupts since all the new Rx packets will be dropped
* The msix handler won't allocate new Rx descriptors so the device won't be
* able to send new packets.
*
* When such a situation is detected - execute rx cleanup task in another thread
*/
static void
check_for_empty_rx_ring(struct ena_adapter *adapter)
{
struct ena_ring *rx_ring;
int i, refill_required;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (!ENA_FLAG_ISSET(ENA_FLAG_DEV_UP, adapter))
Check for Rx ring state to prevent from stall in the ENA driver In case when Rx ring is full and driver will fail to allocate Rx mbufs, the ring could be stalled. Keep alive is checking every second for Rx ring state, and if it is full for two cycles, then trigger rx_cleanup routine in another thread. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12856
2017-11-09 11:48:22 +00:00
return;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET, adapter))
Check for Rx ring state to prevent from stall in the ENA driver In case when Rx ring is full and driver will fail to allocate Rx mbufs, the ring could be stalled. Keep alive is checking every second for Rx ring state, and if it is full for two cycles, then trigger rx_cleanup routine in another thread. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12856
2017-11-09 11:48:22 +00:00
return;
for (i = 0; i < adapter->num_queues; i++) {
rx_ring = &adapter->rx_ring[i];
refill_required = ena_com_free_desc(rx_ring->ena_com_io_sq);
if (unlikely(refill_required == (rx_ring->ring_size - 1))) {
rx_ring->empty_rx_queue++;
if (rx_ring->empty_rx_queue >= EMPTY_RX_REFILL) {
counter_u64_add(rx_ring->rx_stats.empty_rx_ring,
1);
device_printf(adapter->pdev,
"trigger refill for ring %d\n", i);
Lock optimization in ENA Handle IO interrupts using filter routine. That way, the main cleanup task could be moved to the separate thread using taskqueue. The deferred Rx cleanup task was removed, and now the cleanup task is begin called instead. That way, the Rx lock could be removed. In addition, Queue management (wake up and stop TX ring) was added, so the TX cleanup task can be performed mostly lockless. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:29:24 +00:00
taskqueue_enqueue(rx_ring->que->cleanup_tq,
&rx_ring->que->cleanup_task);
Check for Rx ring state to prevent from stall in the ENA driver In case when Rx ring is full and driver will fail to allocate Rx mbufs, the ring could be stalled. Keep alive is checking every second for Rx ring state, and if it is full for two cycles, then trigger rx_cleanup routine in another thread. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12856
2017-11-09 11:48:22 +00:00
rx_ring->empty_rx_queue = 0;
}
} else {
rx_ring->empty_rx_queue = 0;
}
}
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Add notification AENQ handler for ENA Notification AENQ handler is responsible for handling requests from ENA device. Missing Tx threshold, Tx timeout and keep alive timeout can be set using hints from the aenq descriptor which can be delivered in the ENA admin notification. The queue suspending and resuming tasks are not supported by the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:09:53 +00:00
static void ena_update_hints(struct ena_adapter *adapter,
struct ena_admin_ena_hw_hints *hints)
{
struct ena_com_dev *ena_dev = adapter->ena_dev;
if (hints->admin_completion_tx_timeout)
ena_dev->admin_queue.completion_timeout =
hints->admin_completion_tx_timeout * 1000;
if (hints->mmio_read_timeout)
/* convert to usec */
ena_dev->mmio_read.reg_read_to =
hints->mmio_read_timeout * 1000;
if (hints->missed_tx_completion_count_threshold_to_reset)
adapter->missing_tx_threshold =
hints->missed_tx_completion_count_threshold_to_reset;
if (hints->missing_tx_completion_timeout) {
if (hints->missing_tx_completion_timeout ==
ENA_HW_HINTS_NO_TIMEOUT)
adapter->missing_tx_timeout = ENA_HW_HINTS_NO_TIMEOUT;
else
adapter->missing_tx_timeout =
SBT_1MS * hints->missing_tx_completion_timeout;
}
if (hints->driver_watchdog_timeout) {
if (hints->driver_watchdog_timeout == ENA_HW_HINTS_NO_TIMEOUT)
adapter->keep_alive_timeout = ENA_HW_HINTS_NO_TIMEOUT;
else
adapter->keep_alive_timeout =
SBT_1MS * hints->driver_watchdog_timeout;
}
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
static void
ena_timer_service(void *data)
{
struct ena_adapter *adapter = (struct ena_adapter *)data;
struct ena_admin_host_info *host_info =
adapter->ena_dev->host_attr.host_info;
check_for_missing_keep_alive(adapter);
check_for_admin_com_state(adapter);
Check for missing MSI-x and Tx completions in ENA If the first MSI-x won't be executed, then the timer service will detect that and trigger device reset. The checking for missing Tx completion was reworked, so it will also check for missing interrupts. Checking number of missing Tx completions can be performed after loop, instead of checking it every iteration. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:16:56 +00:00
check_for_missing_completions(adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Check for Rx ring state to prevent from stall in the ENA driver In case when Rx ring is full and driver will fail to allocate Rx mbufs, the ring could be stalled. Keep alive is checking every second for Rx ring state, and if it is full for two cycles, then trigger rx_cleanup routine in another thread. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12856
2017-11-09 11:48:22 +00:00
check_for_empty_rx_ring(adapter);
Refactor style of the ENA driver * Change all conditional checks in "if" statement to boolean expressions * Initialize variables with too complex values outside the declaration * Fix indentations * Move code associated with sysctls to ena_sysctl.c file * For consistency, remove unnecesary "return" from void functions * Use if_getdrvflags() function instead of accesing variable directly Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12860
2017-11-09 11:57:02 +00:00
if (host_info != NULL)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_update_host_info(host_info, adapter->ifp);
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
if (unlikely(ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET, adapter))) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "Trigger reset is on\n");
taskqueue_enqueue(adapter->reset_tq, &adapter->reset_task);
return;
}
/*
* Schedule another timeout one second from now.
*/
callout_schedule_sbt(&adapter->timer_service, SBT_1S, SBT_1S, 0);
}
Split Rx/Tx from initialization code in ENA driver Move Rx/Tx routines to separate file. Some functions: * ena_restore_device, * ena_destroy_device, * ena_up, * ena_down, * ena_refill_rx_bufs could be reused in upcoming netmap code in the driver. To make it possible, they were moved to ena.h header. Differential Revision: https://reviews.freebsd.org/D21933 Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:44:26 +00:00
void
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
ena_destroy_device(struct ena_adapter *adapter, bool graceful)
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
{
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
if_t ifp = adapter->ifp;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
struct ena_com_dev *ena_dev = adapter->ena_dev;
bool dev_up;
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
if (!ENA_FLAG_ISSET(ENA_FLAG_DEVICE_RUNNING, adapter))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return;
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
if_link_state_change(ifp, LINK_STATE_DOWN);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
callout_drain(&adapter->timer_service);
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
dev_up = ENA_FLAG_ISSET(ENA_FLAG_DEV_UP, adapter);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
if (dev_up)
ENA_FLAG_SET_ATOMIC(ENA_FLAG_DEV_UP_BEFORE_RESET, adapter);
else
ENA_FLAG_CLEAR_ATOMIC(ENA_FLAG_DEV_UP_BEFORE_RESET, adapter);
if (!graceful)
ena_com_set_admin_running_state(ena_dev, false);
if (ENA_FLAG_ISSET(ENA_FLAG_DEV_UP, adapter))
ena_down(adapter);
/*
* Stop the device from sending AENQ events (if the device was up, and
* the trigger reset was on, ena_down already performs device reset)
*/
if (!(ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET, adapter) && dev_up))
ena_com_dev_reset(adapter->ena_dev, adapter->reset_reason);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Update ena-com HAL to v1.1.4.3 and update driver accordingly The newest ena-com HAL supports LLQv2 and introduces API changes. In order not to break the driver compilation it was updated/fixed in a following way: * Change version of the driver to 0.8.0 * Provide reset cause when triggering reset of the device * Reset device after attach fails * In the reset task free management irq after calling ena_down. Admin queue can still be used before ena_down is called, or when it is being handled * Do not reset device if ena_reset_task fails * Move call of the ena_com_dev_reset to the ena_down() routine - it should be called only if interface was up * Use different function for checking empty space on the sq ring (ena-com API change) * Fix typo on ENA_TX_CLEANUP_THRESHOLD * Change checking for EPERM with EOPNOTSUPP - change in the ena-com API * Minor style fixes Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Amazon.com, Inc. Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12143
2017-10-31 12:41:07 +00:00
ena_free_mgmnt_irq(adapter);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_disable_msix(adapter);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_com_abort_admin_commands(ena_dev);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_com_wait_for_abort_completion(ena_dev);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_com_admin_destroy(ena_dev);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_com_mmio_reg_read_request_destroy(ena_dev);
Update ena-com HAL to v1.1.4.3 and update driver accordingly The newest ena-com HAL supports LLQv2 and introduces API changes. In order not to break the driver compilation it was updated/fixed in a following way: * Change version of the driver to 0.8.0 * Provide reset cause when triggering reset of the device * Reset device after attach fails * In the reset task free management irq after calling ena_down. Admin queue can still be used before ena_down is called, or when it is being handled * Do not reset device if ena_reset_task fails * Move call of the ena_com_dev_reset to the ena_down() routine - it should be called only if interface was up * Use different function for checking empty space on the sq ring (ena-com API change) * Fix typo on ENA_TX_CLEANUP_THRESHOLD * Change checking for EPERM with EOPNOTSUPP - change in the ena-com API * Minor style fixes Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Amazon.com, Inc. Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12143
2017-10-31 12:41:07 +00:00
adapter->reset_reason = ENA_REGS_RESET_NORMAL;
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ENA_FLAG_CLEAR_ATOMIC(ENA_FLAG_TRIGGER_RESET, adapter);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
ENA_FLAG_CLEAR_ATOMIC(ENA_FLAG_DEVICE_RUNNING, adapter);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
static int
ena_device_validate_params(struct ena_adapter *adapter,
struct ena_com_dev_get_features_ctx *get_feat_ctx)
{
if (memcmp(get_feat_ctx->dev_attr.mac_addr, adapter->mac_addr,
ETHER_ADDR_LEN) != 0) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
"Error, mac address are different\n");
return (EINVAL);
}
if (get_feat_ctx->dev_attr.max_mtu < if_getmtu(adapter->ifp)) {
device_printf(adapter->pdev,
"Error, device max mtu is smaller than ifp MTU\n");
return (EINVAL);
}
return 0;
}
Split Rx/Tx from initialization code in ENA driver Move Rx/Tx routines to separate file. Some functions: * ena_restore_device, * ena_destroy_device, * ena_up, * ena_down, * ena_refill_rx_bufs could be reused in upcoming netmap code in the driver. To make it possible, they were moved to ena.h header. Differential Revision: https://reviews.freebsd.org/D21933 Submitted by: Rafal Kozik <rk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:44:26 +00:00
int
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
ena_restore_device(struct ena_adapter *adapter)
{
struct ena_com_dev_get_features_ctx get_feat_ctx;
struct ena_com_dev *ena_dev = adapter->ena_dev;
if_t ifp = adapter->ifp;
device_t dev = adapter->pdev;
int wd_active;
int rc;
ENA_FLAG_SET_ATOMIC(ENA_FLAG_ONGOING_RESET, adapter);
rc = ena_device_init(adapter, dev, &get_feat_ctx, &wd_active);
if (rc != 0) {
device_printf(dev, "Cannot initialize device\n");
goto err;
}
/*
* Only enable WD if it was enabled before reset, so it won't override
* value set by the user by the sysctl.
*/
if (adapter->wd_active != 0)
adapter->wd_active = wd_active;
rc = ena_device_validate_params(adapter, &get_feat_ctx);
if (rc != 0) {
device_printf(dev, "Validation of device parameters failed\n");
goto err_device_destroy;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
rc = ena_handle_updated_queues(adapter, &get_feat_ctx);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
if (rc != 0)
goto err_device_destroy;
ENA_FLAG_CLEAR_ATOMIC(ENA_FLAG_ONGOING_RESET, adapter);
/* Make sure we don't have a race with AENQ Links state handler */
if (ENA_FLAG_ISSET(ENA_FLAG_LINK_UP, adapter))
if_link_state_change(ifp, LINK_STATE_UP);
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rc = ena_enable_msix_and_set_admin_interrupts(adapter,
adapter->num_queues);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
if (rc != 0) {
device_printf(dev, "Enable MSI-X failed\n");
goto err_device_destroy;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
/* If the interface was up before the reset bring it up */
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
if (ENA_FLAG_ISSET(ENA_FLAG_DEV_UP_BEFORE_RESET, adapter)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rc = ena_up(adapter);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
if (rc != 0) {
device_printf(dev, "Failed to create I/O queues\n");
goto err_disable_msix;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
Fix ENA keep-alive timeout due to prolonged reset When the ENA_FLAG_DEVICE_RUNNING flag is disabled, the AENQ handlers aren't executed. To fix that, the watchdog timestamp should be updated just before enabling the watchdog. Timer service was always being enabled, even if the device wasn't up before the reset. That shouldn't happen, as the timer service is being executed only for working interface. Differential Revision: https://reviews.freebsd.org/D21932 Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:39:54 +00:00
/* Indicate that device is running again and ready to work */
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
ENA_FLAG_SET_ATOMIC(ENA_FLAG_DEVICE_RUNNING, adapter);
Fix ENA keep-alive timeout due to prolonged reset When the ENA_FLAG_DEVICE_RUNNING flag is disabled, the AENQ handlers aren't executed. To fix that, the watchdog timestamp should be updated just before enabling the watchdog. Timer service was always being enabled, even if the device wasn't up before the reset. That shouldn't happen, as the timer service is being executed only for working interface. Differential Revision: https://reviews.freebsd.org/D21932 Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:39:54 +00:00
if (ENA_FLAG_ISSET(ENA_FLAG_DEV_UP_BEFORE_RESET, adapter)) {
/*
* As the AENQ handlers weren't executed during reset because
* the flag ENA_FLAG_DEVICE_RUNNING was turned off, the
* timestamp must be updated again That will prevent next reset
* caused by missing keep alive.
*/
adapter->keep_alive_timestamp = getsbinuptime();
callout_reset_sbt(&adapter->timer_service, SBT_1S, SBT_1S,
ena_timer_service, (void *)adapter, 0);
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
device_printf(dev,
"Device reset completed successfully, Driver info: %s\n", ena_version);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
return (rc);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
err_disable_msix:
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_free_mgmnt_irq(adapter);
ena_disable_msix(adapter);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
err_device_destroy:
Fix error handling when ENA reset fails Before the patch, error handling was not releasing all resources and was not issuing device reset if the reset task failed. That could cause memory leak and fault of the device. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:35:43 +00:00
ena_com_abort_admin_commands(ena_dev);
ena_com_wait_for_abort_completion(ena_dev);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_com_admin_destroy(ena_dev);
Fix error handling when ENA reset fails Before the patch, error handling was not releasing all resources and was not issuing device reset if the reset task failed. That could cause memory leak and fault of the device. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:35:43 +00:00
ena_com_dev_reset(ena_dev, ENA_REGS_RESET_DRIVER_INVALID_STATE);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
ena_com_mmio_reg_read_request_destroy(ena_dev);
err:
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ENA_FLAG_CLEAR_ATOMIC(ENA_FLAG_DEVICE_RUNNING, adapter);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
ENA_FLAG_CLEAR_ATOMIC(ENA_FLAG_ONGOING_RESET, adapter);
device_printf(dev, "Reset attempt failed. Can not reset the device\n");
return (rc);
}
static void
ena_reset_task(void *arg, int pending)
{
struct ena_adapter *adapter = (struct ena_adapter *)arg;
if (unlikely(!ENA_FLAG_ISSET(ENA_FLAG_TRIGGER_RESET, adapter))) {
device_printf(adapter->pdev,
"device reset scheduled but trigger_reset is off\n");
return;
}
sx_xlock(&adapter->ioctl_sx);
ena_destroy_device(adapter, false);
ena_restore_device(adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
sx_unlock(&adapter->ioctl_sx);
}
/**
* ena_attach - Device Initialization Routine
* @pdev: device information struct
*
* Returns 0 on success, otherwise on failure.
*
* ena_attach initializes an adapter identified by a device structure.
* The OS initialization, configuring of the adapter private structure,
* and a hardware reset occur.
**/
static int
ena_attach(device_t pdev)
{
struct ena_com_dev_get_features_ctx get_feat_ctx;
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
struct ena_llq_configurations llq_config;
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
struct ena_calc_queue_size_ctx calc_queue_ctx = { 0 };
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
static int version_printed;
struct ena_adapter *adapter;
struct ena_com_dev *ena_dev = NULL;
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
const char *queue_type_str;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
int io_queue_num;
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
int rid, rc;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
adapter = device_get_softc(pdev);
adapter->pdev = pdev;
mtx_init(&adapter->global_mtx, "ENA global mtx", NULL, MTX_DEF);
sx_init(&adapter->ioctl_sx, "ENA ioctl sx");
/* Set up the timer service */
callout_init_mtx(&adapter->timer_service, &adapter->global_mtx, 0);
adapter->keep_alive_timeout = DEFAULT_KEEP_ALIVE_TO;
adapter->missing_tx_timeout = DEFAULT_TX_CMP_TO;
adapter->missing_tx_max_queues = DEFAULT_TX_MONITORED_QUEUES;
adapter->missing_tx_threshold = DEFAULT_TX_CMP_THRESHOLD;
if (version_printed++ == 0)
device_printf(pdev, "%s\n", ena_version);
/* Allocate memory for ena_dev structure */
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
ena_dev = malloc(sizeof(struct ena_com_dev), M_DEVBUF,
M_WAITOK | M_ZERO);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
adapter->ena_dev = ena_dev;
ena_dev->dmadev = pdev;
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
rid = PCIR_BAR(ENA_REG_BAR);
adapter->memory = NULL;
adapter->registers = bus_alloc_resource_any(pdev, SYS_RES_MEMORY,
&rid, RF_ACTIVE);
if (unlikely(adapter->registers == NULL)) {
device_printf(pdev,
"unable to allocate bus resource: registers!\n");
rc = ENOMEM;
goto err_dev_free;
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_dev->bus = malloc(sizeof(struct ena_bus), M_DEVBUF,
M_WAITOK | M_ZERO);
/* Store register resources */
((struct ena_bus*)(ena_dev->bus))->reg_bar_t =
rman_get_bustag(adapter->registers);
((struct ena_bus*)(ena_dev->bus))->reg_bar_h =
rman_get_bushandle(adapter->registers);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(((struct ena_bus*)(ena_dev->bus))->reg_bar_h == 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(pdev, "failed to pmap registers bar\n");
rc = ENXIO;
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
goto err_bus_free;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
ena_dev->tx_mem_queue_type = ENA_ADMIN_PLACEMENT_POLICY_HOST;
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
/* Initially clear all the flags */
ENA_FLAG_ZERO(adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Device initialization */
rc = ena_device_init(adapter, pdev, &get_feat_ctx, &adapter->wd_active);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(pdev, "ENA device init failed! (err: %d)\n", rc);
rc = ENXIO;
goto err_bus_free;
}
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
set_default_llq_configurations(&llq_config);
rc = ena_set_queues_placement_policy(pdev, ena_dev, &get_feat_ctx.llq,
&llq_config);
if (unlikely(rc != 0)) {
device_printf(pdev, "failed to set placement policy\n");
goto err_com_free;
}
if (ena_dev->tx_mem_queue_type == ENA_ADMIN_PLACEMENT_POLICY_HOST)
queue_type_str = "Regular";
else
queue_type_str = "Low Latency";
device_printf(pdev, "Placement policy: %s\n", queue_type_str);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
adapter->keep_alive_timestamp = getsbinuptime();
adapter->tx_offload_cap = get_feat_ctx.offload.tx;
memcpy(adapter->mac_addr, get_feat_ctx.dev_attr.mac_addr,
ETHER_ADDR_LEN);
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
calc_queue_ctx.ena_dev = ena_dev;
calc_queue_ctx.get_feat_ctx = &get_feat_ctx;
calc_queue_ctx.pdev = pdev;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* calculate IO queue number to create */
io_queue_num = ena_calc_io_queue_num(adapter, &get_feat_ctx);
ENA_ASSERT(io_queue_num > 0, "Invalid queue number: %d\n",
io_queue_num);
adapter->num_queues = io_queue_num;
Read max MTU from the ENA device The device now provides driver with max available MTU value it can handle. The function setting MTU for the interface was simplified and reworked to follow up this changes. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12870
2017-11-09 13:35:07 +00:00
adapter->max_mtu = get_feat_ctx.dev_attr.max_mtu;
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
// Set the requested Rx ring size
adapter->rx_ring_size = ENA_DEFAULT_RING_SIZE;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* calculatre ring sizes */
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
rc = ena_calc_queue_size(adapter, &calc_queue_ctx);
if (unlikely((rc != 0) || (io_queue_num <= 0))) {
rc = EFAULT;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
goto err_com_free;
}
Update ena-com HAL to v1.1.4.3 and update driver accordingly The newest ena-com HAL supports LLQv2 and introduces API changes. In order not to break the driver compilation it was updated/fixed in a following way: * Change version of the driver to 0.8.0 * Provide reset cause when triggering reset of the device * Reset device after attach fails * In the reset task free management irq after calling ena_down. Admin queue can still be used before ena_down is called, or when it is being handled * Do not reset device if ena_reset_task fails * Move call of the ena_com_dev_reset to the ena_down() routine - it should be called only if interface was up * Use different function for checking empty space on the sq ring (ena-com API change) * Fix typo on ENA_TX_CLEANUP_THRESHOLD * Change checking for EPERM with EOPNOTSUPP - change in the ena-com API * Minor style fixes Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Amazon.com, Inc. Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12143
2017-10-31 12:41:07 +00:00
adapter->reset_reason = ENA_REGS_RESET_NORMAL;
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
adapter->tx_ring_size = calc_queue_ctx.tx_queue_size;
adapter->rx_ring_size = calc_queue_ctx.rx_queue_size;
adapter->max_tx_sgl_size = calc_queue_ctx.max_tx_sgl_size;
adapter->max_rx_sgl_size = calc_queue_ctx.max_rx_sgl_size;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
adapter->buf_ring_size = ENA_DEFAULT_BUF_RING_SIZE;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* set up dma tags for rx and tx buffers */
rc = ena_setup_tx_dma_tag(adapter);
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
if (unlikely(rc != 0)) {
device_printf(pdev, "Failed to create TX DMA tag\n");
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
goto err_com_free;
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
rc = ena_setup_rx_dma_tag(adapter);
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
if (unlikely(rc != 0)) {
device_printf(pdev, "Failed to create RX DMA tag\n");
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
goto err_tx_tag_free;
Rework printouts and logging level in ENA driver The driver was printing out a lot of information upon failure, which does not have to be interested for the user. Changing logging level required to rebuild driver with proper flags. The proper sysctl was added, so the level now can be changed dynamically using bitmask. Levels of printouts were adjusted to keep on mind end user instead of debugging purposes. More verbose messages were added to align the driver with the Linux. Fix building error introduced by the r325506 by casting csum_flags to uint64_t. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12868
2017-11-09 13:30:39 +00:00
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* initialize rings basic information */
Add tuneable drbr ring size and hw queues depth for ENA The driver now supports per adapter tuning of buffer ring size and HW Rx ring size. It can be achieved using sysctl node dev.ena.X. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:28:03 +00:00
device_printf(pdev,
"Creating %d io queues. Rx queue size: %d, Tx queue size: %d\n",
io_queue_num,
calc_queue_ctx.rx_queue_size,
calc_queue_ctx.tx_queue_size);
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
ena_init_io_rings(adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Change attach order to prevent crash upon failure in ENA The if_detach was causing crash if the MSI-x configuration in the attach failed. To prevent this issue, the ifnet is being configured at the end of the attach function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:24:47 +00:00
rc = ena_enable_msix_and_set_admin_interrupts(adapter, io_queue_num);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Change attach order to prevent crash upon failure in ENA The if_detach was causing crash if the MSI-x configuration in the attach failed. To prevent this issue, the ifnet is being configured at the end of the attach function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:24:47 +00:00
device_printf(pdev,
"Failed to enable and set the admin interrupts\n");
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
goto err_io_free;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Change attach order to prevent crash upon failure in ENA The if_detach was causing crash if the MSI-x configuration in the attach failed. To prevent this issue, the ifnet is being configured at the end of the attach function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:24:47 +00:00
/* setup network interface */
rc = ena_setup_ifnet(pdev, adapter, &get_feat_ctx);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0)) {
Change attach order to prevent crash upon failure in ENA The if_detach was causing crash if the MSI-x configuration in the attach failed. To prevent this issue, the ifnet is being configured at the end of the attach function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:24:47 +00:00
device_printf(pdev, "Error with network interface setup\n");
goto err_msix_free;
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
Add error handling to the ENA driver if init of the reset task fails Also, to simplify cleaning routine, reset task is initialized before allocating statistics and other resources. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10925
2017-05-30 11:56:54 +00:00
/* Initialize reset task queue */
TASK_INIT(&adapter->reset_task, 0, ena_reset_task, adapter);
adapter->reset_tq = taskqueue_create("ena_reset_enqueue",
M_WAITOK | M_ZERO, taskqueue_thread_enqueue, &adapter->reset_tq);
taskqueue_start_threads(&adapter->reset_tq, 1, PI_NET,
"%s rstq", device_get_nameunit(adapter->pdev));
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Initialize statistics */
ena_alloc_counters((counter_u64_t *)&adapter->dev_stats,
sizeof(struct ena_stats_dev));
Rework counting of hardware statistics in ENA driver Do not read all statistics from the device, instead count them in the driver except from RX drops - they are received directly from the NIC in the AENQ descriptor. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: imp Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12852
2017-10-31 16:31:23 +00:00
ena_alloc_counters((counter_u64_t *)&adapter->hw_stats,
sizeof(struct ena_hw_stats));
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_sysctl_add_nodes(adapter);
Introduce NETMAP support in ENA Mock implementation of NETMAP routines is located in ena_netmap.c/.h files. All code is protected under the DEV_NETMAP macro. Makefile was updated with files and flag. As ENA driver provide own implementations of (un)likely it must be undefined before including NETMAP headers. ena_netmap_attach function is called on the end of NIC attach. It fills structure with NIC configuration and callbacks. Then provides it to netmap_attach. Similarly netmap_detach is called during ena_detach. Three callbacks are used. nm_register is implemented by ena_netmap_reg. It is called when user space application open or close NIC in NETMAP mode. Current action is recognized based on onoff parameter: true means on and false off. As NICs rings need to be reconfigured ena_down and ena_up are reused. When user space application wants to receive new packets from NIC nm_rxsync is called, and when there are new packets ready for Tx nm_txsync is called. Differential Revision: https://reviews.freebsd.org/D21934 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:51:18 +00:00
#ifdef DEV_NETMAP
rc = ena_netmap_attach(adapter);
if (rc != 0) {
device_printf(pdev, "netmap attach failed: %d\n", rc);
goto err_detach;
}
#endif /* DEV_NETMAP */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Tell the stack that the interface is not active */
if_setdrvflagbits(adapter->ifp, IFF_DRV_OACTIVE, IFF_DRV_RUNNING);
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ENA_FLAG_SET_ATOMIC(ENA_FLAG_DEVICE_RUNNING, adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (0);
Introduce NETMAP support in ENA Mock implementation of NETMAP routines is located in ena_netmap.c/.h files. All code is protected under the DEV_NETMAP macro. Makefile was updated with files and flag. As ENA driver provide own implementations of (un)likely it must be undefined before including NETMAP headers. ena_netmap_attach function is called on the end of NIC attach. It fills structure with NIC configuration and callbacks. Then provides it to netmap_attach. Similarly netmap_detach is called during ena_detach. Three callbacks are used. nm_register is implemented by ena_netmap_reg. It is called when user space application open or close NIC in NETMAP mode. Current action is recognized based on onoff parameter: true means on and false off. As NICs rings need to be reconfigured ena_down and ena_up are reused. When user space application wants to receive new packets from NIC nm_rxsync is called, and when there are new packets ready for Tx nm_txsync is called. Differential Revision: https://reviews.freebsd.org/D21934 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:51:18 +00:00
#ifdef DEV_NETMAP
err_detach:
ether_ifdetach(adapter->ifp);
#endif /* DEV_NETMAP */
Change attach order to prevent crash upon failure in ENA The if_detach was causing crash if the MSI-x configuration in the attach failed. To prevent this issue, the ifnet is being configured at the end of the attach function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:24:47 +00:00
err_msix_free:
ena_com_dev_reset(adapter->ena_dev, ENA_REGS_RESET_INIT_ERR);
ena_free_mgmnt_irq(adapter);
ena_disable_msix(adapter);
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
err_io_free:
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_free_all_io_rings_resources(adapter);
ena_free_rx_dma_tag(adapter);
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
err_tx_tag_free:
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_free_tx_dma_tag(adapter);
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
err_com_free:
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_com_admin_destroy(ena_dev);
ena_com_delete_host_info(ena_dev);
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
ena_com_mmio_reg_read_request_destroy(ena_dev);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
err_bus_free:
free(ena_dev->bus, M_DEVBUF);
ena_free_pci_resources(adapter);
Add support for the LLQv2 and WC in ENA LLQ (Low Latency Queue) is the feature, that allows pushing header directly to the device through PCI before even DMA is triggered. It reduces latency, because device can start preparing packet before payload is sent through DMA. To speed up sending data through PCI, the Write Combining is enabled, which allows hardware to buffer data before sending them on the PCI - it allows to reduce number of PCI IO operations. ENAv2 is using special descriptor for the negotiation of the LLQ. Currently, only the default configuration is supported. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:30:52 +00:00
err_dev_free:
free(ena_dev, M_DEVBUF);
Fix ENA driver error handling in attach and basic style fixes The patch contains following changes: * In conditional checks, always check for NULL or 0 instead of negating values * Use malloc and free explicitely, instead of ENA_MEM_FREE and ENA_MEM_FREE (the dmadev passed to macro is never used, and could be a little misleading) * Always check for NULL after calling malloc (few checks were missing) * Rework naming of the goto tags in ena_attach() for consistency * Fix error handling in ena_attach() - few goto instructions were leading to the wrong tag * Destroy MMIO req read request if attach failed * Remove checking for NULL after calling malloc with M_WAITOK flag Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12853
2017-11-07 13:20:41 +00:00
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
return (rc);
}
/**
* ena_detach - Device Removal Routine
* @pdev: device information struct
*
* ena_detach is called by the device subsystem to alert the driver
* that it should release a PCI device.
**/
static int
ena_detach(device_t pdev)
{
struct ena_adapter *adapter = device_get_softc(pdev);
struct ena_com_dev *ena_dev = adapter->ena_dev;
int rc;
/* Make sure VLANS are not using driver */
if (adapter->ifp->if_vlantrunk != NULL) {
device_printf(adapter->pdev ,"VLAN is in use, detach first\n");
return (EBUSY);
}
Change order of ifp release on ENA detach In rare case, when the ifconfig is called just before kldunload, it is possible, that ena_up routine will be called after queue locks are released. To prevent that, ifp is detached before the last ena_down is called and further, the ifp is freed at the end of the function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:22:53 +00:00
ether_ifdetach(adapter->ifp);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/* Free reset task and callout */
callout_drain(&adapter->timer_service);
while (taskqueue_cancel(adapter->reset_tq, &adapter->reset_task, NULL))
taskqueue_drain(adapter->reset_tq, &adapter->reset_task);
taskqueue_free(adapter->reset_tq);
Add locks before each ena_up and ena_down Lock only ena_up and ena_down calls in ioctl handler, instead of whole ioctl. Locking ioctl with sx lock that is sleepable, is not allowed in some cases, e.g. when multicast options are being changed. Additional locking was added in deatch function to prevent race condition with ioctl function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10924
2017-05-30 11:55:02 +00:00
sx_xlock(&adapter->ioctl_sx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_down(adapter);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
ena_destroy_device(adapter, true);
Add locks before each ena_up and ena_down Lock only ena_up and ena_down calls in ioctl handler, instead of whole ioctl. Locking ioctl with sx lock that is sleepable, is not allowed in some cases, e.g. when multicast options are being changed. Additional locking was added in deatch function to prevent race condition with ioctl function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10924
2017-05-30 11:55:02 +00:00
sx_unlock(&adapter->ioctl_sx);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Introduce NETMAP support in ENA Mock implementation of NETMAP routines is located in ena_netmap.c/.h files. All code is protected under the DEV_NETMAP macro. Makefile was updated with files and flag. As ENA driver provide own implementations of (un)likely it must be undefined before including NETMAP headers. ena_netmap_attach function is called on the end of NIC attach. It fills structure with NIC configuration and callbacks. Then provides it to netmap_attach. Similarly netmap_detach is called during ena_detach. Three callbacks are used. nm_register is implemented by ena_netmap_reg. It is called when user space application open or close NIC in NETMAP mode. Current action is recognized based on onoff parameter: true means on and false off. As NICs rings need to be reconfigured ena_down and ena_up are reused. When user space application wants to receive new packets from NIC nm_rxsync is called, and when there are new packets ready for Tx nm_txsync is called. Differential Revision: https://reviews.freebsd.org/D21934 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:51:18 +00:00
#ifdef DEV_NETMAP
netmap_detach(adapter->ifp);
#endif /* DEV_NETMAP */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_free_all_io_rings_resources(adapter);
Rework counting of hardware statistics in ENA driver Do not read all statistics from the device, instead count them in the driver except from RX drops - they are received directly from the NIC in the AENQ descriptor. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: imp Obtained from: Semihalf Sponsored by: Amazon.com, Inc. Differential Revision: https://reviews.freebsd.org/D12852
2017-10-31 16:31:23 +00:00
ena_free_counters((counter_u64_t *)&adapter->hw_stats,
sizeof(struct ena_hw_stats));
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
ena_free_counters((counter_u64_t *)&adapter->dev_stats,
sizeof(struct ena_stats_dev));
rc = ena_free_rx_dma_tag(adapter);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
"Unmapped RX DMA tag associations\n");
rc = ena_free_tx_dma_tag(adapter);
Cover ENA driver code with branch predictioning statements The driver was using it in only few places, so the rest of the code was covered with those statement. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: rlibby Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12861
2017-11-09 11:59:21 +00:00
if (unlikely(rc != 0))
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev,
"Unmapped TX DMA tag associations\n");
ena_free_irqs(adapter);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
ena_free_pci_resources(adapter);
Destroy admin queue after freeing interrupts in ENA driver On heavy load, when interrupt handling routine was slowed down, there could appear memory corruption, because resources were destroyed and interrupt was still being handled. Submitted by: Michal Krawczyk <mk@semihalf.com> Reviewed by: byenduri_gmail.com Obtained from: Semihalf Sponsored by: Amazon, Inc. Differential Revision: https://reviews.freebsd.org/D12858
2017-11-09 11:52:52 +00:00
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
if (likely(ENA_FLAG_ISSET(ENA_FLAG_RSS_ACTIVE, adapter)))
ena_com_rss_destroy(ena_dev);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
ena_com_delete_host_info(ena_dev);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
mtx_destroy(&adapter->global_mtx);
sx_destroy(&adapter->ioctl_sx);
Change order of ifp release on ENA detach In rare case, when the ifconfig is called just before kldunload, it is possible, that ena_up routine will be called after queue locks are released. To prevent that, ifp is detached before the last ena_down is called and further, the ifp is freed at the end of the function. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:22:53 +00:00
if_free(adapter->ifp);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
if (ena_dev->bus != NULL)
free(ena_dev->bus, M_DEVBUF);
if (ena_dev != NULL)
free(ena_dev, M_DEVBUF);
return (bus_generic_detach(pdev));
}
/******************************************************************************
******************************** AENQ Handlers *******************************
*****************************************************************************/
/**
* ena_update_on_link_change:
* Notify the network interface about the change in link status
**/
static void
ena_update_on_link_change(void *adapter_data,
struct ena_admin_aenq_entry *aenq_e)
{
struct ena_adapter *adapter = (struct ena_adapter *)adapter_data;
struct ena_admin_aenq_link_change_desc *aenq_desc;
int status;
if_t ifp;
aenq_desc = (struct ena_admin_aenq_link_change_desc *)aenq_e;
ifp = adapter->ifp;
status = aenq_desc->flags &
ENA_ADMIN_AENQ_LINK_CHANGE_DESC_LINK_STATUS_MASK;
if (status != 0) {
device_printf(adapter->pdev, "link is UP\n");
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ENA_FLAG_SET_ATOMIC(ENA_FLAG_LINK_UP, adapter);
Split ENA reset routine into restore and destroy stages For alignment with Linux driver and better handling ena_detach(), the reset is now calling ena_device_restore() and ena_device_destroy(). The ena_device_destroy() is also being called on ena_detach(), so the code will be more readable. The watchdog is now being activated after reset only, if it was active before. There were added additional checks to ensure, that there is no race with the link state change AENQ handler. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:39:25 +00:00
if (!ENA_FLAG_ISSET(ENA_FLAG_ONGOING_RESET, adapter))
if_link_state_change(ifp, LINK_STATE_UP);
} else {
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
device_printf(adapter->pdev, "link is DOWN\n");
if_link_state_change(ifp, LINK_STATE_DOWN);
Use bitfield for storing global ENA device states As the ENA can have multiple states turned on/off, it is more convenient to store them in single bitfield instead of multiple boolean variables. The bitset FreeBSD API was used for the bitfield implementation, as it provides flexible structure together with API which also supports atomic bitfield operations. For better readability basic macros from API were wrapped into custom ENA_FLAG_* macros, which are filling up common parameters for all calls. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:37:15 +00:00
ENA_FLAG_CLEAR_ATOMIC(ENA_FLAG_LINK_UP, adapter);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
}
Add notification AENQ handler for ENA Notification AENQ handler is responsible for handling requests from ENA device. Missing Tx threshold, Tx timeout and keep alive timeout can be set using hints from the aenq descriptor which can be delivered in the ENA admin notification. The queue suspending and resuming tasks are not supported by the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:09:53 +00:00
static void ena_notification(void *adapter_data,
struct ena_admin_aenq_entry *aenq_e)
{
struct ena_adapter *adapter = (struct ena_adapter *)adapter_data;
struct ena_admin_ena_hw_hints *hints;
ENA_WARN(aenq_e->aenq_common_desc.group != ENA_ADMIN_NOTIFICATION,
"Invalid group(%x) expected %x\n", aenq_e->aenq_common_desc.group,
ENA_ADMIN_NOTIFICATION);
switch (aenq_e->aenq_common_desc.syndrom) {
case ENA_ADMIN_UPDATE_HINTS:
hints =
(struct ena_admin_ena_hw_hints *)(&aenq_e->inline_data_w4);
ena_update_hints(adapter, hints);
break;
default:
device_printf(adapter->pdev,
"Invalid aenq notification link state %d\n",
aenq_e->aenq_common_desc.syndrom);
}
}
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/**
* This handler will called for unknown event group or unimplemented handlers
**/
static void
Print information when ENA admin error occurs ENA_ADMIN_FATAL_ERROR and ENA_ADMIN_WARNING aenq groups were indicated as supported, so the unimplemented_aenq_handler() will print out error message, whenever an error will occur within the ENA admin context. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:08:00 +00:00
unimplemented_aenq_handler(void *adapter_data,
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
struct ena_admin_aenq_entry *aenq_e)
{
Print information when ENA admin error occurs ENA_ADMIN_FATAL_ERROR and ENA_ADMIN_WARNING aenq groups were indicated as supported, so the unimplemented_aenq_handler() will print out error message, whenever an error will occur within the ENA admin context. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:08:00 +00:00
struct ena_adapter *adapter = (struct ena_adapter *)adapter_data;
device_printf(adapter->pdev,
"Unknown event was received or event with unimplemented handler\n");
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
}
static struct ena_aenq_handlers aenq_handlers = {
.handlers = {
[ENA_ADMIN_LINK_CHANGE] = ena_update_on_link_change,
Add notification AENQ handler for ENA Notification AENQ handler is responsible for handling requests from ENA device. Missing Tx threshold, Tx timeout and keep alive timeout can be set using hints from the aenq descriptor which can be delivered in the ENA admin notification. The queue suspending and resuming tasks are not supported by the driver. Submitted by: Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-05-30 13:09:53 +00:00
[ENA_ADMIN_NOTIFICATION] = ena_notification,
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
[ENA_ADMIN_KEEP_ALIVE] = ena_keep_alive_wd,
},
.unimplemented_handler = unimplemented_aenq_handler
};
/*********************************************************************
* FreeBSD Device Interface Entry Points
*********************************************************************/
static device_method_t ena_methods[] = {
/* Device interface */
DEVMETHOD(device_probe, ena_probe),
DEVMETHOD(device_attach, ena_attach),
DEVMETHOD(device_detach, ena_detach),
DEVMETHOD_END
};
static driver_t ena_driver = {
"ena", ena_methods, sizeof(struct ena_adapter),
};
devclass_t ena_devclass;
DRIVER_MODULE(ena, pci, ena_driver, ena_devclass, 0, 0);
Remove stray space from PNP string.
2018-07-08 23:12:27 +00:00
MODULE_PNP_INFO("U16:vendor;U16:device", pci, ena, ena_vendor_info_array,
Reapply, with minor tweaks, r338025, from the original commit: Remove unused and easy to misuse PNP macro parameter Inspired by r338025, just remove the element size parameter to the MODULE_PNP_INFO macro entirely. The 'table' parameter is now required to have correct pointer (or array) type. Since all invocations of the macro already had this property and the emitted PNP data continues to include the element size, there is no functional change. Mostly done with the coccinelle 'spatch' tool: $ cat modpnpsize0.cocci @normaltables@ identifier b,c; expression a,d,e; declarer MODULE_PNP_INFO; @@ MODULE_PNP_INFO(a,b,c,d, -sizeof(d[0]), e); @singletons@ identifier b,c,d; expression a; declarer MODULE_PNP_INFO; @@ MODULE_PNP_INFO(a,b,c,&d, -sizeof(d), 1); $ rg -l MODULE_PNP_INFO -- sys | \ xargs spatch --in-place --sp-file modpnpsize0.cocci (Note that coccinelle invokes diff(1) via a PATH search and expects diff to tolerate the -B flag, which BSD diff does not. So I had to link gdiff into PATH as diff to use spatch.) Tinderbox'd (-DMAKE_JUST_KERNELS). Approved by: re (glen)
2018-09-26 17:12:14 +00:00
nitems(ena_vendor_info_array) - 1);
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
MODULE_DEPEND(ena, pci, 1, 1, 1);
MODULE_DEPEND(ena, ether, 1, 1, 1);
Introduce NETMAP support in ENA Mock implementation of NETMAP routines is located in ena_netmap.c/.h files. All code is protected under the DEV_NETMAP macro. Makefile was updated with files and flag. As ENA driver provide own implementations of (un)likely it must be undefined before including NETMAP headers. ena_netmap_attach function is called on the end of NIC attach. It fills structure with NIC configuration and callbacks. Then provides it to netmap_attach. Similarly netmap_detach is called during ena_detach. Three callbacks are used. nm_register is implemented by ena_netmap_reg. It is called when user space application open or close NIC in NETMAP mode. Current action is recognized based on onoff parameter: true means on and false off. As NICs rings need to be reconfigured ena_down and ena_up are reused. When user space application wants to receive new packets from NIC nm_rxsync is called, and when there are new packets ready for Tx nm_txsync is called. Differential Revision: https://reviews.freebsd.org/D21934 Submitted by: Rafal Kozik <rk@semihalf.com> Michal Krawczyk <mk@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon, Inc.
2019-10-31 15:51:18 +00:00
#ifdef DEV_NETMAP
MODULE_DEPEND(ena, netmap, 1, 1, 1);
#endif /* DEV_NETMAP */
Add support for Amazon Elastic Network Adapter (ENA) NIC ENA is a networking interface designed to make good use of modern CPU features and system architectures. The ENA device exposes a lightweight management interface with a minimal set of memory mapped registers and extendable command set through an Admin Queue. The driver supports a range of ENA devices, is link-speed independent (i.e., the same driver is used for 10GbE, 25GbE, 40GbE, etc.), and has a negotiated and extendable feature set. Some ENA devices support SR-IOV. This driver is used for both the SR-IOV Physical Function (PF) and Virtual Function (VF) devices. ENA devices enable high speed and low overhead network traffic processing by providing multiple Tx/Rx queue pairs (the maximum number is advertised by the device via the Admin Queue), a dedicated MSI-X interrupt vector per Tx/Rx queue pair, and CPU cacheline optimized data placement. The ENA driver supports industry standard TCP/IP offload features such as checksum offload and TCP transmit segmentation offload (TSO). Receive-side scaling (RSS) is supported for multi-core scaling. The ENA driver and its corresponding devices implement health monitoring mechanisms such as watchdog, enabling the device and driver to recover in a manner transparent to the application, as well as debug logs. Some of the ENA devices support a working mode called Low-latency Queue (LLQ), which saves several more microseconds. This feature will be implemented for driver in future releases. Submitted by: Michal Krawczyk <mk@semihalf.com> Jakub Palider <jpa@semihalf.com> Jan Medala <jan@semihalf.com> Obtained from: Semihalf Sponsored by: Amazon.com Inc. Differential revision: https://reviews.freebsd.org/D10427
2017-05-22 14:46:13 +00:00
/*********************************************************************/
Reference in New Issue Copy Permalink