7f87c0406d
- Ever since the workaround for the silicon bug of TSO4 causing MAC hangs was committed in r295133, CSUM_TSO always got disabled unconditionally by em(4) on the first invocation of em_init_locked(). However, even with that problem fixed, it turned out that for at least e. g. 82579 not all necessary TSO workarounds are in place, still causing MAC hangs even at Gigabit speed. Thus, for stable/11, TSO usage was deliberately disabled in r323292 (r323293 for stable/10) for the EM-class by default, allowing users to turn it on if it happens to work with their particular EM MAC in a Gigabit-only environment. In head, the TSO workaround for speeds other than Gigabit was lost with the conversion to iflib(9) in r311849 (possibly along with another one or two TSO workarounds). Yet at the same time, for EM-class MACs TSO4 got enabled by default again, causing device hangs. Therefore, change the default for this hardware class back to have TSO4 off, allowing users to turn it on manually if it happens to work in their environment as we do in stable/{10,11}. An alternative would be to add a whitelist of EM-class devices where TSO4 actually is reliable with the workarounds in place, but given that the advantage of TSO at Gigabit speed is rather limited - especially with the overhead of these workarounds -, that's really not worth it. [1] This change includes the addition of an isc_capabilities to struct if_softc_ctx so iflib(9) can also handle interface capabilities that shouldn't be enabled by default which is used to handle the default-off capabilities of e1000 as suggested by shurd@ and moving their handling from em_setup_interface() to em_if_attach_pre() accordingly. - Although 82543 support TSO4 in theory, the former lem(4) didn't have support for TSO4, presumably because TSO4 is even more broken in the LEM-class of MACs than the later EM ones. Still, TSO4 for LEM-class devices was enabled as part of the conversion to iflib(9) in r311849, causing device hangs. So revert back to the pre-r311849 behavior of not supporting TSO4 for LEM-class at all, which includes not creating a TSO DMA tag in iflib(9) for devices not having IFCAP_TSO4 set. [2] - In fact, the FreeBSD TCP stack can handle a TSO size of IP_MAXPACKET (65535) rather than FREEBSD_TSO_SIZE_MAX (65518). However, the TSO DMA must have a maxsize of the maximum TSO size plus the size of a VLAN header for software VLAN tagging. The iflib(9) converted em(4), thus, first correctly sets scctx->isc_tx_tso_size_max to EM_TSO_SIZE in em_if_attach_pre(), but later on overrides it with IP_MAXPACKET in em_setup_interface() (apparently, left-over from pre-iflib(9) times). So remove the later and correct iflib(9) to correctly cap the maximum TSO size reported to the stack at IP_MAXPACKET. While at it, let iflib(9) use if_sethwtsomax*(). This change includes the addition of isc_tso_max{seg,}size DMA engine constraints for the TSO DMA tag to struct if_shared_ctx and letting iflib_txsd_alloc() automatically adjust the maxsize of that tag in case IFCAP_VLAN_MTU is supported as requested by shurd@. - Move the if_setifheaderlen(9) call for adjusting the maximum Ethernet header length from {ixgbe,ixl,ixlv,ixv,em}_setup_interface() to iflib(9) so adjustment is automatically done in case IFCAP_VLAN_MTU is supported. As a consequence, this adjustment now is also done in case of bnxt(4) which missed it previously. - Move the reduction of the maximum TSO segment count reported to the stack by the number of m_pullup(9) calls (which in the worst case, can add another mbuf and, thus, the requirement for another DMA segment each) in the transmit path for performance reasons from em_setup_interface() to iflib_txsd_alloc() as these pull-ups are now done in iflib_parse_header() rather than in the no longer existing em_xmit(). Moreover, this optimization applies to all drivers using iflib(9) and not just em(4); all in-tree iflib(9) consumers still have enough room to handle full size TSO packets. Also, reduce the adjustment to the maximum number of m_pullup(9)'s now performed in iflib_parse_header(). - Prior to the conversion of em(4)/igb(4)/lem(4) and ixl(4) to iflib(9) in r311849 and r335338 respectively, these drivers didn't enable IFCAP_VLAN_HWFILTER by default due to VLAN events not being passed through by lagg(4). With iflib(9), IFCAP_VLAN_HWFILTER was turned on by default but also lagg(4) was fixed in that regard in r203548. So just remove the now redundant and defunct IFCAP_VLAN_HWFILTER handling in {em,ixl,ixlv}_setup_interface(). - Nuke other redundant IFCAP_* setting in {em,ixl,ixlv}_setup_interface() which is (more completely) already done in {em,ixl,ixlv}_if_attach_pre() now. - Remove some redundant/dead setting of scctx->isc_tx_csum_flags in em_if_attach_pre(). - Remove some IFCAP_* duplicated either directly or indirectly (e. g. via IFCAP_HWCSUM) in {EM,IGB,IXL}_CAPS. - Don't bother to fiddle with IFCAP_HWSTATS in ixgbe(4)/ixgbev(4) as iflib(9) adds that capability unconditionally. - Remove some unused macros from em(4). - Bump __FreeBSD_version as some of the above changes require the modules of drivers using iflib(9) to be recompiled. Okayed by: sbruno@ at 201806 DevSummit Transport Working Group [1] Reviewed by: sbruno (earlier version), erj PR: 219428 (part of; comment #10) [1], 220997 (part of; comment #3) [2] Differential Revision: https://reviews.freebsd.org/D15720 |
||
---|---|---|
.. | ||
i40e_adminq_cmd.h | ||
i40e_adminq.c | ||
i40e_adminq.h | ||
i40e_alloc.h | ||
i40e_common.c | ||
i40e_dcb.c | ||
i40e_dcb.h | ||
i40e_devids.h | ||
i40e_hmc.c | ||
i40e_hmc.h | ||
i40e_lan_hmc.c | ||
i40e_lan_hmc.h | ||
i40e_nvm.c | ||
i40e_osdep.c | ||
i40e_osdep.h | ||
i40e_prototype.h | ||
i40e_register.h | ||
i40e_status.h | ||
i40e_type.h | ||
if_ixl.c | ||
if_ixlv.c | ||
ixl_debug.h | ||
ixl_iw_int.h | ||
ixl_iw.c | ||
ixl_iw.h | ||
ixl_pf_i2c.c | ||
ixl_pf_iov.c | ||
ixl_pf_iov.h | ||
ixl_pf_main.c | ||
ixl_pf_qmgr.c | ||
ixl_pf_qmgr.h | ||
ixl_pf.h | ||
ixl_txrx.c | ||
ixl.h | ||
ixlv_vc_mgr.h | ||
ixlv.h | ||
ixlvc.c | ||
README | ||
virtchnl.h |
ixl FreeBSD* Base Driver and ixlv VF Driver for the Intel XL710 Ethernet Controller Family /*$FreeBSD$*/ ================================================================ August 26, 2014 Contents ======== - Overview - Supported Adapters - The VF Driver - Building and Installation - Additional Configurations - Known Limitations Overview ======== This file describes the IXL FreeBSD* Base driver and the IXLV VF Driver for the XL710 Ethernet Family of Adapters. The Driver has been developed for use with FreeBSD 10.0 or later, but should be compatible with any supported release. For questions related to hardware requirements, refer to the documentation supplied with your Intel XL710 adapter. All hardware requirements listed apply for use with FreeBSD. Supported Adapters ================== The drivers in this release are compatible with XL710 and X710-based Intel Ethernet Network Connections. SFP+ Devices with Pluggable Optics ---------------------------------- SR Modules ---------- Intel DUAL RATE 1G/10G SFP+ SR (bailed) FTLX8571D3BCV-IT Intel DUAL RATE 1G/10G SFP+ SR (bailed) AFBR-703SDZ-IN2 LR Modules ---------- Intel DUAL RATE 1G/10G SFP+ LR (bailed) FTLX1471D3BCV-IT Intel DUAL RATE 1G/10G SFP+ LR (bailed) AFCT-701SDZ-IN2 QSFP+ Modules ------------- Intel TRIPLE RATE 1G/10G/40G QSFP+ SR (bailed) E40GQSFPSR Intel TRIPLE RATE 1G/10G/40G QSFP+ LR (bailed) E40GQSFPLR QSFP+ 1G speed is not supported on XL710 based devices. X710/XL710 Based SFP+ adapters support all passive and active limiting direct attach cables that comply with SFF-8431 v4.1 and SFF-8472 v10.4 specifications. The VF Driver ================== The VF driver is normally used in a virtualized environment where a host driver manages SRIOV, and provides a VF device to the guest. With this first release the only host environment tested was using Linux QEMU/KVM. Support is planned for Xen and VMWare hosts at a later time. In the FreeBSD guest the IXLV driver would be loaded and will function using the VF device assigned to it. The VF driver provides most of the same functionality as the CORE driver, but is actually a slave to the Host, access to many controls are actually accomplished by a request to the Host via what is called the "Admin queue". These are startup and initialization events however, once in operation the device is self-contained and should achieve near native performance. Some notable limitations of the VF environment: for security reasons the driver is never permitted to be promiscuous, therefore a tcpdump will not behave the same with the interface. Second, media info is not available from the PF, so it will always appear as auto. Tarball Building and Installation ========================= NOTE: You must have kernel sources installed to compile the driver tarball. These instructions assume a standalone driver tarball, building the driver already in the kernel source is simply a matter of adding the device entry to the kernel config file, or building in the ixl or ixlv module directory. In the instructions below, x.x.x is the driver version as indicated in the name of the driver tarball. The example is for ixl, the same procedure applies for ixlv. 1. Move the base driver tar file to the directory of your choice. For example, use /home/username/ixl or /usr/local/src/ixl. 2. Untar/unzip the archive: tar xfz ixl-x.x.x.tar.gz 3. To install man page: cd ixl-x.x.x gzip -c ixl.4 > /usr/share/man/man4/ixl.4.gz 4. To load the driver onto a running system: cd ixl-x.x.x/src make load 5. To assign an IP address to the interface, enter the following: ifconfig ixl<interface_num> <IP_address> 6. Verify that the interface works. Enter the following, where <IP_address> is the IP address for another machine on the same subnet as the interface that is being tested: ping <IP_address> 7. If you want the driver to load automatically when the system is booted: cd ixl-x.x.x/src make make install Edit /boot/loader.conf, and add the following line: if_ixl_load="YES" Edit /etc/rc.conf, and create the appropriate ifconfig_ixl<interface_num> entry: ifconfig_ixl<interface_num>="<ifconfig_settings>" Example usage: ifconfig_ixl0="inet 192.168.10.1 netmask 255.255.255.0" NOTE: For assistance, see the ifconfig man page. Configuration and Tuning ========================= Both drivers supports Transmit/Receive Checksum Offload for IPv4 and IPv6, TSO forIPv4 and IPv6, LRO, and Jumbo Frames on all 40 Gigabit adapters. Jumbo Frames ------------ To enable Jumbo Frames, use the ifconfig utility to increase the MTU beyond 1500 bytes. - The Jumbo Frames setting on the switch must be set to at least 22 byteslarger than that of the adapter. - The maximum MTU setting for Jumbo Frames is 9706. This value coincides with the maximum jumbo frames size of 9728. To modify the setting, enter the following: ifconfig ixl<interface_num> <hostname or IP address> mtu 9000 - To confirm an interface's MTU value, use the ifconfig command. To confirm the MTU used between two specific devices, use: route get <destination_IP_address> VLANs ----- To create a new VLAN pseudo-interface: ifconfig <vlan_name> create To associate the VLAN pseudo-interface with a physical interface and assign a VLAN ID, IP address, and netmask: ifconfig <vlan_name> <ip_address> netmask <subnet_mask> vlan <vlan_id> vlandev <physical_interface> Example: ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ixl0 In this example, all packets will be marked on egress with 802.1Q VLAN tags, specifying a VLAN ID of 10. To remove a VLAN pseudo-interface: ifconfig <vlan_name> destroy Checksum Offload ---------------- Checksum offloading supports IPv4 and IPv6 with TCP and UDP packets and is supported for both transmit and receive. Checksum offloading for transmit and recieve is enabled by default for both IPv4 and IPv6. Checksum offloading can be enabled or disabled using ifconfig. Transmit and receive offloading for IPv4 and Ipv6 are enabled and disabled seperately. NOTE: TSO requires Tx checksum, so when Tx checksum is disabled, TSO will also be disabled. To enable Tx checksum offloading for ipv4: ifconfig ixl<interface_num> txcsum4 To disable Tx checksum offloading for ipv4: ifconfig ixl<interface_num> -txcsum4 (NOTE: This will disable TSO4) To enable Rx checksum offloading for ipv6: ifconfig ixl<interface_num> rxcsum6 To disable Rx checksum offloading for ipv6: ifconfig ixl<interface_num> -rxcsum6 (NOTE: This will disable TSO6) To confirm the current settings: ifconfig ixl<interface_num> TSO --- TSO supports both IPv4 and IPv6 and is enabled by default. TSO can be disabled and enabled using the ifconfig utility. NOTE: TSO requires Tx checksum, so when Tx checksum is disabled, TSO will also be disabled. To disable TSO IPv4: ifconfig ixl<interface_num> -tso4 To enable TSO IPv4: ifconfig ixl<interface_num> tso4 To disable TSO IPv6: ifconfig ixl<interface_num> -tso6 To enable TSO IPv6: ifconfig ixl<interface_num> tso6 To disable BOTH TSO IPv4 and IPv6: ifconfig ixl<interface_num> -tso To enable BOTH TSO IPv4 and IPv6: ifconfig ixl<interface_num> tso LRO --- Large Receive Offload is enabled by default. It can be enabled or disabled by using the ifconfig utility. NOTE: LRO should be disabled when forwarding packets. To disable LRO: ifconfig ixl<interface_num> -lro To enable LRO: ifconfig ixl<interface_num> lro Flow Control (IXL only) ------------ Flow control is disabled by default. To change flow control settings use sysctl. To enable flow control to Rx pause frames: sysctl dev.ixl.<interface_num>.fc=1 To enable flow control to Tx pause frames: sysctl dev.ixl.<interface_num>.fc=2 To enable flow control to Rx and Tx pause frames: sysctl dev.ixl.<interface_num>.fc=3 To disable flow control: sysctl dev.ixl.<interface_num>.fc=0 NOTE: You must have a flow control capable link partner. NOTE: The VF driver does not have access to flow control, it must be managed from the host side. Important system configuration changes: ======================================= -Change the file /etc/sysctl.conf, and add the line: hw.intr_storm_threshold: 0 (the default is 1000) -Best throughput results are seen with a large MTU; use 9706 if possible. -The default number of descriptors per ring is 1024, increasing this may improve performance depending on the use case. -The VF driver uses a relatively large buf ring, this was found to eliminate UDP transmit errors, it is a tuneable, and if no UDP traffic is used it can be reduced. It is memory used per queue. Known Limitations ================= Network Memory Buffer allocation -------------------------------- FreeBSD may have a low number of network memory buffers (mbufs) by default. If your mbuf value is too low, it may cause the driver to fail to initialize and/or cause the system to become unresponsive. You can check to see if the system is mbuf-starved by running 'netstat -m'. Increase the number of mbufs by editing the lines below in /etc/sysctl.conf: kern.ipc.nmbclusters kern.ipc.nmbjumbop kern.ipc.nmbjumbo9 kern.ipc.nmbjumbo16 kern.ipc.nmbufs The amount of memory that you allocate is system specific, and may require some trial and error. Also, increasing the follwing in /etc/sysctl.conf could help increase network performance: kern.ipc.maxsockbuf net.inet.tcp.sendspace net.inet.tcp.recvspace net.inet.udp.maxdgram net.inet.udp.recvspace UDP Stress Test Dropped Packet Issue ------------------------------------ Under small packet UDP stress test with the ixl driver, the FreeBSD system may drop UDP packets due to the fullness of socket buffers. You may want to change the driver's Flow Control variables to the minimum value for controlling packet reception. Disable LRO when routing/bridging --------------------------------- LRO must be turned off when forwarding traffic. Lower than expected performance ------------------------------- Some PCIe x8 slots are actually configured as x4 slots. These slots have insufficient bandwidth for full line rate with dual port and quad port devices. In addition, if you put a PCIe Generation 3-capable adapter into a PCIe Generation 2 slot, you cannot get full bandwidth. The driver detects this situation and writes the following message in the system log: "PCI-Express bandwidth available for this card is not sufficient for optimal performance. For optimal performance a x8 PCI-Express slot is required." If this error occurs, moving your adapter to a true PCIe Generation 3 x8 slot will resolve the issue. Support ======= For general information and support, go to the Intel support website at: http://support.intel.com If an issue is identified with the released source code on the supported kernel with a supported adapter, email the specific information related to the issue to freebsdnic@mailbox.intel.com. License ======= This software program is released under the terms of a license agreement between you ('Licensee') and Intel. Do not use or load this software or any associated materials (collectively, the 'Software') until you have carefully read the full terms and conditions of the LICENSE located in this software package. By loadingor using the Software, you agree to the terms of this Agreement. If you do not agree with the terms of this Agreement, do not install or use the Software. * Other names and brands may be claimed as the property of others.